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IMAGE PROCESSING APPARATUS 

The present invention relates to the field of image processing, and more 
particularly to the processing of image data defining a plurality of images 
recorded at different positions and orientations relative to a scene to determine 
camera projections for the images. 

A camera projection for an image of a scene comprises a mathematical 
definition (typically a matrix or tensor) defining how points in the scene are 
projected into the image by the camera which recorded the image. 
Accordingly, a camera projection defines a mapping between a three- 
dimensional space containing the camera (typically referred to as the "world 
coordinate system") and a two-dimensional space of the image plane. 
Examples of common camera projections are given in "Epipolar Geometry in 
Stereo, Motion and Object Recognition" by Xu and Zhang, Chapter 2, Kluwer 
Academic Press, ISBN 0792341996. Examples include the perspective 
projection, the orthographic projection, the weak perspective projection, the 
affine projection, etc. 

The combination of a camera projection with data defining the intrinsic 
parameters of the camera which recorded the image (that is, focal length, 
image aspect ratio, first order radial distortion coefficient, skew angle - the 
angle between the axes of the pixel grid, and principal point - the point at 
which the camera optical axis intersects the viewing plane) defines a position 
and orientation for the camera when the image was recorded. This position 
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and orientation is defined in terms of a rotation and translation of the camera 
in the world coordinate system. In the case of some types of camera 
projection, such as a perspective projection, the recording position and 
orientation of the camera is completely specified by the camera projection and 
camera intrinsic parameters. For other types of camera projection, such as an 
affme projection, the recording position and orientation is defmed by the 
camera projections and camera intrinsic parameters up to certain limits. For 
example, in the case of an affme projection, one limit is that the translation of 
the camera in the "Z" (depth) direction in the world coordinate system is not 
defmed; this is because die camera would have recorded the same image for 
all translations in the depth direction for an afOne projection and accordingly 
a single translation cannot be determined. 

As is well known, a camera projection for an image can be calculated without 
knowing the intrinsic camera parameters. Further, if required, and if some or 
all of the intrinsic parameters are not known, they can be calculated from a 
plurality of images of the scene and the associated calculated camera 
projections. 

A number of techniques are known for calculating camera projections for 
images of a scene by processing data defining the images. 

For example, EP-A-0898245 discloses a technique in which a camera 
projection is calculated for each image in a sequence by considering the 
images in respective overlapping groups in the sequence, each group 
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comprising three images. More particularly, camera projections are calculated 
for images 1, 2 and 3 in the sequence, then images 2, 3 and 4, followed by 
images 3, 4 and 5 etc. until camera projections have been calculated for all the 
images in the sequence. 

"Calibration of Image Sequences for Model Visualisation" by Broadhurst and 
CipoUa in Proceedings of the 1999 IEEE Computer Society Conference on 
Computer Vision and Pattem Recognition, Volume 1, ISBN 0769501494 
discloses a technique in which the trifocal tensor of the three most extreme 
positional views in a long sequence of video images of a scene is calculated, 
as this is more accurate than the tensor of three successive views. Once this 
outer tensor is known, projection matrices for the intermediate frames are 
calculated. An iterative algorithm using Levenberg-Marquardt niininiisation 
is then employed to perturb the twelve entries of the last camera matrix so that 
the algebraic error along the whole sequence is minimised, 

"Multi-View 3D Estimation and Applications to Match Move" by Sawhney et 
al in 1999 IEEE Workshop on Multi-View Modelling and Analysis of Visual 
Scenes, ISBN 0769501 109 discloses a technique ux which the positions and 
orientations of each image in a sequence of images are initially calculated by 
pairwise estimation. The sequence of images is then split into a plurality of 
sub-sequences with a few frames overlap between consecutive sub-sequences, 
and the initial pairwise estimates are used to create position and orientation 
estimates for each image which are consistent over the sub-sequence in which 
the image lies. Subsequently, the sub-sequences are stitched together by using 
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points that are visible in two overlapping sub-sequences to represent both the 
sub-sequences in a common coordinate system. In a final step, the positions 
and orientations for the complete set of images is bundle adjusted to compute 
die maximum likelihood estimate of the recording positions and orientations. 

Despite the known techniques for calculating camera projections, there is still 
a requirement for techniques with improved efficiency (that is, processing 
resources and time necessary to carry out the technique) and/or which improve 
the accuracy of the calculated solutions. 

The present invention has been made with this in mind. 

According to the present invention, there is provided a method or apparatus for 
processing input data defining a sequence of images to calculate camera 
projections for images within the sequence, in which a plurality of groups of 
images are selected fi:om the sequence in dependence upon the input data, each 
group containing at least two images, a set is formed firom each group by 
calculating relative camera projections for the images in the group, at least 
some of the sets are expanded by calcidating a camera projection for a fiirther 
image in the sequence relative to the set, and the relationship between the 
camera projections for dififerent sets is detemuned. 

The present invention also provides a computer program product, embodied 
for example as a storage device or signal, including instructions for causing a 
programmable processing apparatus to become operable to perform such a 
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method or to become configured as such an apparatus. 

Embodiments of the invention will now be described, by way of example only, 
with reference to the accompanying drawings, in which: 

Figure 1 schematically shows the components of a modular system in which 
the present invention is embodied; 

Figure 2 schematically shows the components of an embodiment of the 
invention, together with the notional functional processing units into which the 
processing apparatus component may become configured when programmed 
by programming instructions; 

Figure 3 shows the processing operations performed on input data by the 
apparatus shown in Figure 2; 

Figure 4 schematically illustrates a sequence of images recorded at different 
positions and orientations around an object, which is input to the processing 
apparatus shown in Figure 1; 

Figure 5 shows the processing operations performed at step S4 in Figure 3 ; 

Figure 6 shows the processing operations performed at step S24 and step S34 
in Figure 5; 
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Figure 7 shows the processing operations performed at step S44 in Figure 5; 

Figures 8a and 8b schematically illustrate an example of the selection of 
keyframes at step S4 in Figure 3 and the selection of triples of keyframes at 
step S6 in Figure 3; 

Figure 9 shows the processing operations performed at step S6 in Figure 3; 

Figure 10 shows the processing operations performed at step S8 in Figure 3; 

Figure 1 1 schematically illustrates an example of a result after the processing 
at step S8 in Figure 3 has been applied to the example shown in Figure 8b; 

Figure 12 shows the processing operations performed at step SIO in Figure 3; 

Figure 13 shows the processing operations performed at step S204 in 
Figure 12; 

Figure 14 shows an example of a list of correspondences generated at step 
S224 in Figure 13; 

Figures 15a, 15b, 15c and 15d schematically illustrate the processing 
performed at step S206 in Figure 12; 

Figiure 16 shows the processing operations performed at step/S206 in 
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Figure 12; 

Figure 17 shows an example of data read at step S270 in Figure 16; 

Figure 18 schematically illustrates an example of a result after the processing 
at step SIO in Figure 3 has been applied to the example shown in Figure 11; 

Figure 19 shows the processing operations performed at step S12 in Figure 3; 

Figure 20 shows the processing operations performed at step S290 in 
Figure 19; 

Figure 21 schematically illustrates an example of a result after the processing 
at step S290 in Figure 19 has been appUed to the example shown in Figure 18; 

Figure 22 shows the processing operations performed at step S292 in 
Figure 19; 

Figure 23 shows the processing operations performed at step S332 in 
Figure 22; 

Figure 24 schematically illustrates an example of a result after the processing 
at step S292 in Figure 19 has been applied to the example shown in Figure 21; 

Figure 25 shows the processing operations performed at step S294;^ ' 
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Figxire 26 schematically illustrates a result of applying the processing at step 
S294 in Figure 19 to the example shown in Figure 24; and 

Figure 27 schematically shows an example of a result of applying the 
processing at steps S4 to S12 in Figure 3 to the example shown in Figure 4. 

The components of a modular system in which the present invention is 
embodied are schematically shown in Figure 1. 

These components can be effected as processor-implemented instructions, 
hardware or a combination thereof. 

Referring to Figure 1, the components are arranged to process data defining 
images (still or moving) of one or more objects in order to generate data 
defining a three-dimensional computer model of the object(s). 

The input image data may be received in a variety of ways, such as directly 
from one or more digital cameras, via a storage device such as a disk or CD 
ROM, by digitisation of photographs using a scanner, or by downloading 
image data from a database, for example via a datalink such as the Internet, 
etc. 

The generated 3D model data may be used to: display an image of the 
object(s) from a desired viewing position; control manufacturing equipment 
to manufacture a-model of the object(s), for example by controlling cutting 
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apparatus to cut material to the appropriate dimensions; perform processing to 
recognise the object(s), for example by comparing it to data stored in a 
database; carry out processing to measure the object(s), for example by taking 
absolute measurements to record the size of the object(s), or by comparing the 
model with models of the object(s) previously generated to determine changes 
therebetween; carry out processing so as to control a robot to navigate around 
the object(s); store information in a geographic information system (GIS) or 
other topographic database; or transmit the object data representing the model 
to a remote processing device for any such processing, either on a storage 
device or as a signal (for example, the data may be transmitted in virtual 
reality modelling language (VRML) format over the Intemet, enabling it to be 
processed by a WWW browser); etc. 

The feature detection and matching module 2 is arranged to receive image data 
recorded by a still camera from different positions relative to the object(s) (the 
different positions being achieved by moving the camera and/or the object(s)). 
The received data is then processed in order to match features within the 
different images (that is, to identify points in the images which correspond to 
the same physical point on the object(s)). 

The feature detection and tracking module 4 is arranged to receive image data 
recorded by a video camera as the relative positions of the camera and 
object(s) are changed (by moving the video camera and/or the object(s)). As 
in the feature detection and matching module 2, the feature detection and 
tracking module 4 detects features, such as comers, in the images. However, 
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the feature detection and tiacking module 4 then tracks the detected features 
between frames of image data in order to determine the positions of the 
features in other images. 

The camera position calculation module 6 is arranged to use the features 
matched across images by the feature detection and matching module 2 or the 
feature detection and tracking module 4 to calculate a respective camera 
projection for each image and to calculate the transformation between the 
camera positions at which the images were recorded and hence determine the 
orientation and position of the camera focal plane when each image was 
recorded. 

The feature detection and matching module 2 and the camera position 
calculation module 6 may be arranged to perform processing in an iterative 
maimer. That is, using camera positions and orientations calctilated by the 
camera position calculation module 6, the feature detection and matching 
module 2 may detect and match further features in the images using epipolar 
geometry in a conventional manner, and the further matched features may then 
be used by the camera position calculation module 6 to recalculate the camera 
positions and orientations. 

If the positions at which the images were recorded are already known, then, 
as indicated by arrow 8 in Figure 1, the image data need not be processed by 
the feature detection and matching module 2, the feature detection and tracking 
module 4, or the camera position^ calculation module 6, : For example, the 
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images may be recorded by momating a number of cameras on a calibrated rig 
arranged to hold the cameras in known positions relative to the object(s). 

Alternatively, it is possible to determine the positions of a plm^ality of cameras 
5 relative to the object(s) by adding calibration markers to the object(s) and 
calculating the positions of the cameras from the positions of the calibration 
markers in images recorded by the cameras. The calibration markers may 
comprise pattems of light projected onto the object(s). Camera calibration 
module 10 is therefore provided to receive image data from a plurality of 

10 cameras at fixed positions showing the object(s) together with calibration 
markers, and to process the data to determine the positions of the cameras. A 
preferred method of calculating the positions of the cameras (and also internal 
parameters of each camera, such as the focal length etc) is described in 
"Calibrating and 3D Modelling with a Multi-Camera System" by Wiles and 

15 Davison in 1999 IEEE Workshop on Multi-View Modelling and Analysis of 
Visual Scenes, ISBN 0769501 109. 

The 3D object surface generation module 12 is arranged to receive image data 
showing the object(s) and data defining the positions at which the images were 
20 recorded, and to process the data to generate a 3D computer model 
representing the actual surface(s) of the object(s), such as a polygon mesh 
model. 

The texture data generation module 14 is arranged to generate texture data for 
25 rendering onto the surface model produced by the 3D object surface generation 
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module 12. The texture data is generated from the input image data showing 
the object(s). 

Techniques that can be used to perform the processing in the modules shown 
in Figure 1 are described in EP-A-0898245, EP-A-0901105, pending US 
appUcations 09/129077, 09/129079 and 09/129080, the full contents of which 
are incorporated herein by cross-reference, and also Aimex A, 

The present invention is embodied in particular as part of the camera position 
calculation module 6, and provides particular advantage when there is a large 
number of input images, for example images recorded by a video camera. 
Accordingly, a description will now be given of the feature detection and 
tracking modiile 4 and the camera position calculation module 6. 

To assist understanding, the processing operations performed by the feature 
detection and tracking module 4 and the camera position calculation module 
6 in the embodiment will be described with reference to functional xmits. 

Figure 2 shows examples of such functional units and their interconnections 
within a single: processing apparatus 20 which is arranged to perform the 
processing operations of the feature detection and tracking module 4 and the 
camera position calculation module 6. 

In this embodiment, processing apparatus 20 is a conventional processing 
apparatus, such as a personal computer, containing,- in a conventional maimer, 
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one or more processors, memory, graphics cards etc together with a display 
device 22, such as a conventional personal computer monitor, and user input 
devices 24, such as a keyboard, mouse etc. 

The processing apparatus 20 is programmed to operate in accordance with 
programming instructions input, for example, as data stored on a data storage 
medium, such as disk 26, and/or as a signal 28 input to the processing 
apparatus, for example from a remote database, by transmission over a 
conununication network (not shown) such as the Internet or by transmission 
through the atmosphere, and/or entered by a user via a user input device 24 
such as a keyboard. 

When programmed by the programming instructions, the processing apparatus 
20 effectively becomes configured into a number of functional units for 
performing the processing operations which will be described below. 

As noted above, examples of such functional units and their interconnections 
are shown in Figure 2. The xmits and intercormections illustrated in Figure 2 
are, however, notional and are shown for illustration purposes only to assist 
imderstanding; they do not necessarily represent the exact rniits and 
connections into which the processor, memory etc of the processing apparatus 
20 become configured. 

Referring to the functional units shown in Figure 2, a central controller 30 
processes inputs from the user input devices 24, and also provides control and 
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processing for a number of the other functional units. Memory 32 is provided 
for use by central controller 30 and the other functional units. 

Data store 40 stores input data input to the processing apparatus 20 as data 
stored on a storage device, such as disk 42, or as a signal 44 transmitted to the 
processing apparatus 20. In this embodiment, the input data defines a 
sequence of video images (frames) of an object, together with the intrinsic 
parameters of the camera which recorded the images, that is, aspect ratio, focal 
length, principal point (the point at which the optical axis intersects the 
imaging plane), first order radial distortion coefficient, and skew (the angle 
between the axes on the pixel grid; because the axes may not be exactly 
orthogonal). 

Feature detector and tracker 50 processes the input image data to detect 
features, such as comers, in the images and to track the position of each 
detected feature between the input image frames. 

Keyframe selector 60 uses information from feature detector and tracker 50 
about the tracked features in each frame to select a sub-set of the input image 
frames as "keyframes", and groups the selected keyframes for fiirther 
processing. 

Camera transformation calculator 70 calculates camera projections for 
different ones of the keyframes selected by keyframe selector 60. Further, in 
this embodiment, .camera transformation calculation 70 is arranged to use the . 
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intrinsic camera parameters stored in data store 40 with the calculated camera 
projections to calculate transformations defining the relationship between the 
positions and orientations at which the selected keyframes were recorded. 
Camera transformation calculator 70 is arranged to calculate transformations 
in accordance with a number of different procedures, control instructions for 
which are stored in procedure instructions store 80. 

Feature matcher 90 matches features between different ones of the keyframes 
selected by keyframe selector 60 to provide additional data for use by camera 
transformation calculator 70 in calculating the camera transformations. 

Camera transformation store 100 stores camera transformations calculated by 
camera transformation calculator 70, and controls the output of these 
transformations as output data, for example as data on a storage device, such 
as disk 102, or as a signal 104. 

Display processor 110, under the control of central controller 30, displays 
images and instructions to a user via display device 22. 

Figure 3 shows the processing operations performed by processing apparatus 
20 in this embodiment. 

Referring to Figtire 3, at step S2, image data defining images of a scene 
recorded at different locations input to the processing apparatus 20, for 
example on disk 42 or as a signal 44, is stored in the data store 40. 
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Referring to Figure 4, in which an example scene is shown (in this case, a 
museum statue 120), the data stored at step S2 defines a plurality of image 
frames 130 showing the statue 120 from different positions and/or viewing 
directions, the image fi*ames 130 being recorded by moving video camera 140 
in the direction of arrow 150 around the statue 120, 

In addition, data is stored at step S2 defining the intrinsic parameters of the 
camera 140, that is the focal length, the aspect ratio (height-to-width) of the 
pixels 130, the principal point, the first order radial distortion coefficient, and 
the skew angle. This data, too, may be input on a storage device such as disk 
42 or as a signal 44, or, instead, may be input by a user using an input device 
24 such as a keyboard. 

At step S4, feature detector and tracker 50 and keyfirame selector 60 together 
with central controller 30 process the data stored at step S2 to identify feature 
points in the input image firames, to track the positions of the feature points 
across different input image fi:ames, and to select a sub-set of the input image 
fi*ames as keyfirames. 

Figure 5 shows the processing operations performed at step S4 in this 
embodiment. 

Referring to Figure 5, at step S20, central controller 30 reads the number of 
input fi-ames and sets a threshold value for use in. subsequent processing to 
reduce the number of image firames. for which camera transformations are to 
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be calculated. More particularly, the threshold value is set in dependence 
upon the number of input frames, the processing power available within 
processing apparatus 20 and any constraints on the time available within which 
camera transformations for the input image frames must be calculated. As will 
be described below, the threshold value set at step S20 represents the 
minimum number of frames which must be present between the keyframes 
selected by keyframe selector 60. Accordingly, if the threshold value is set to 
"zero", then keyframes may be selected which are successive image frames in 
the input sequence 130, whereas if a threshold value of "ten" is set, then 
subsequent keyframe selection will result in the number of input frames being 
reduced by a factor of at least ten. 

At step S22, feature detector and tracker 50 processes the image data for the 
first image frame in the sequence 130 (that is, the frame recorded at the 
earliest time because the frames are arranged in time-of-recording order vnthin 
sequence 130) to identify features in the image for tracking. In this 
embodiment, processing is carried out in a conventional manner, for example 
as described in section 1 of Aimex A, to identify comer points in the image 
frame for tracking. 

At step S24, feature detector and tracker 50 together with central controller 30 
perform tracking of the feature points identified at step S22 for the next frame 
in the input sequence 130. 

In this embodiment, the tracking process uses a constant , image velocity 
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Kalman filter to predict the motion of the feature points, and a correlation- 
based matcher to make measurements of feature point correspondences 
between image frames. The method assumes that the motion of feature points 
is smooth enough across the sequence of image frames 130 that a constant 
velocity Kalman filter is useful, and that feature point measurements and 
motion can be modelled by Gaussians. 

Figure 6 shows the processing operations performed by feature detector and 
tracker 50 and central controller 30 at step S24 to perform tracking for one 
feature point (the same processing being performed for each of the other 
feature points). 

Referring to Figure 6, at step S70, feature detector and tracker 50 reads the 
current 4D state, X, of the feature point (defined by the position and velocity 
vector of the feature point in a conventional maimer) and the state covaiiance 
matrix, K, for the feature point. In this embodiment, the initial value for the 
4D state of a feature point is defined using the position of the feature point in 
the first image frame and zero velocity, while the initial condition for the state 
covariance matrix, K^, is defined by: 

fO 0) 

where is the initial velocity variance and is set to = 200(pixels/frame)^, 
and "I" is the identity matrix. 
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At step S72, feature detector and tracker 50 uses the parameter values read at 
step S70 in the following standard Kalman filter equations to predict the value 
of the 4D state and state covariance matrix for the feature point for the nevs^ 
image frame (that is the next frame in the sequence 130), assuming a constant 
5 velocity and remdom uniform Gaussian acceleration model for the dynamics: 

^„.,= • -(2) 

^KQ^-Q -(3) 

where: 

10 X„^.i is the prediction of the 4D state of the feature point for the new frame; 

X„ is the 4D state of the feature point for the current frame; 

( K^i is the prediction of the state covariance matrix for the new frame; 

15 

Kn is the state covariance matrix for the current frame; 

6 is the transition matrix which, in this embodiment, is constant and has the 
following value: 
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Q is the process covariance matrix which, in this embodiment, is constant and 
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has the following value; 

fo 0) 

Q- ^ 2, -.-(5) 

where is the process velocity variance which, in this embodiment is set to 
the fixed value of 200 (pixels/frame)^ if the camera 140 was hand-held when 
the images 130 were recorded, and to the value of 50 (pixels/frame)^ if the 
camera 140 was mounted on a support, such as a robot arm etc, to record the 
images 130. The value of a^^50 (pixels/frame)^ would also be applicable if 
the camera 140 had been held stationary (for example on a tripod) while the 
object 120 was rotated on a turntable. 



At step S74, feature detector and tracker 50 tests whether the predicted 
positional uncertainty of the feature point (given by the top two diagonal 
elements of the state covariance matrix, K^j) is greater than a threshold which, 
in this embodiment is set to 400 pixelsl 

If it is determined at step S74 that the predicted positional uncertainty is 
greater than the threshold, then, at step S76, the feature point is discarded so 
that it is no longer tracked. 

On the other hand, if it is determined at step S74 tiiat the predicted positional 
uncertainty is not greater than the threshold, then, at step S78, feature detector 
and tracker 50 searches within a window of the new frame (that is, frame n+1) 
to identify pixels which match the feature' poi^^ tracked. More 
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particularly, feature detector and trackc 50 searches pixels in the new frame 
within a rectcuigular window centred on the predicted position of the feature 
point in the new frame, the dimensions of the rectangular window being given 



where and K22 are the top two diagonal elements of the state co variance 
matrix, K^^. 

A respective correlation score is calculated for each pixel within the window 
in the new image defining the correlation between the pixel and the pixels 
within a second window of size 15 pixels by 15 pixels centred on the last 
measured position of the feature point in a frame. This may be a frame earlier 
in the sequence 130 than frame "n" since a position may not be found in each 
frame for the feature point. Accordingly, the pixels in the second window 
around the last measured position of- the feature point are cached to avoid 
having to retrieve the whole image from memory to allow the correlation score 
to be calculated. 

In this embodiment, the correlation score, C, is calculated between the pixels 
in the 15 by 15 window centred on the last measured position of the feature 
point and the pixels in a 15 by 15 pixel window centred on the pixel in the 
new frame for which a correlation score is to be calculated (that is, a pixel 
from the window of size Ax by Ay in the new frame centred on the predicted 
position). More particularly, the correlation score is calculated in a 



by: 




....(6) 
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conventional manner, by calculating: 



where: 



^''' = j^i:u!"-<i'-»? ....(8) 



5 "i" labels the different pixels in the window being considered; 
"j" labels the different images (j=i,2); 
I"^ is the pixel image intensity in the fnst unage of the pair; 

10 

F is the pixel image intensity in the second image of the pair; 

o denotes the mean value of the image intensity in the window; 

15 N is the number of pixels in the window (15x 15=225 in this embodiment). 

In this embodiment, feature detector and tracker 50 calculates the value of 
a<'>' and o^^)' given by equation (8) before calculating the correlation 
score, C, given by equation(7). Further, onlyifthe values ofboth and o^^^' 

20 

are greater than 100 does feanire detector and tracker 50 calculate a 
correlation score, C. This ensures that correlation scores are not calculated 
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based on windows of pixels which have approximately uniform texture 
(which would lead to artificially high correlation scores). 

At step S80, feature detector and tracker 50 uses the correlation scores 
calculated at step S78 to determine whether a match has been found for the 
feature point being tracked. More particularly, feature detector and tracker 50 
reads the values of the highest two correlation scores calculated at step S78. 
Feature detector and tracker 50 then determines whether, firstly, the top 
correlation score is larger than a threshold Co (in this embodiment set to 
Co = 0,9), and, secondly, the difference between the top two correlation scores 
is larger than a threshold AC (in this embodiment AC = 0.001). If these first 
and second criteria are met, then it is determined that the pixel having the 
highest correlation score represents a match for the feature point being tracked. 
On the other hand, if the two criteria are not met, then it is determined that no 
match has been found. 

If it is determined at step S80 that a match has been found then, at step S82, 
the measurement of the feature point position in the new firame made at step 
S78 (that is, the pixel having the highest correlation score) is used to update 
the 4D state, X, of the feature point, the state covariance matrix, K, and the 
Kalman filter gain, G, in a conventional maimer as follows: 



G = KH\HKH^^R) 



-1 



--(9) 



....(10) 
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K-iI-GH)K ....(U) 

A 

where X is the position measurement (pixel position) from step S78, "H" is the 
measurement matrix, and "R" is the measurement co variance matrix. In this 
embodiment, "H" is constant, being given by: 



R is given by: 



^ = (^0) ....(12) 



-(13) 



where is the measurement variance given by: 

= {rk^a) . .(14) 

where k = \/(Kn K22) (that is, a measure of the positional uncertainty), "r" is 
a parameter related to the likelihood of obtaining an outlier (in this 
embodiment set to r = 0. 1), and "a" is a parameter related to the measurement 
uncertainty of inliers (in this embodiment set to a=1.0). 

The updated parameters given by equations (9), (10) and (11) are stored by 
feature detector and tracker 50, but die values of the parameters before they 
were updated are also retained as they may be required for subsequent 
processing at step S108 or SI 10 in Figure 7, which is described later. 
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On the other hand, if it is determined at step S80 that a match has not been 
found for the feature point being tracked, then, at step S84, the feature detector 
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and tracker 50 updates the 4D state, X, of the feature point, the state 
covariance matrix, K, and the Kahnan filter gain in a conventional manner as 
follows: 

G = 0 ....(15) 
5 X-X ....(16) 

K-K ...,(17) 

Referring again to Figure 5, at step S26, keyframe selector 60 calculates a 
score representing a total positional uncertainty for the tracked features La the 
new frame (that is, the frame for which the positions of the features were 
10 predicted and measured at step S24). More particularly, in this embodiment, 
keyframe selector 60 calculates an uncertainty score as follows: 



Uncertainty score = min(af,a^3j ....(18) 

n lowest 
measurement 
variances 



where: o\ is the measurement variance of the position of the "i"th 

1 5 feature point (comer in this embodiment) given by equation ( 14) 

above; 

represents an upper threshold on the measurement variance, 
and is set to 40 in this embodiment; 

20 ' ^ ' * ' * ' 
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min (or, a; J is the minimum of a? and ol^^ (so that the 
maximum variance value that a point can contribute to the 
summation is o^J; and 

the summation is carried out for the lowest "n" measurement 
variances ("n" being set to 100 in this embodiment). If there 
are less than "n" tracked feature points for the frame, then the 
uncertainty score is calculated by adding the values min (o-, 
oLc) for the "m" existing points and (n-m) x (that is, (n- 
m)x40 in this embodiment). 

At step S28, keyframe selector 60 determines whether the uncertainty score 
calculated at step S26 is less than a threshold value, which, in this embodiment 
is set to 2,000. 

If it is determined at step S28 that the uncertainty score is not less than the 
threshold, then steps S24 to S28 are iteratively repeated for each subsequent 
frame in the sequence 130 until it is determined at step S28 that the uncertainty 
score for a frame is less than the threshold value. 

When it is determined at step S28 that the uncertainty score is less than a 
threshold value, then, at step S30, keyframe selector 60 designated the frame 
which has the uncertainty score less than the tiu-eshold as a keyframe. 



step S32, feature detector and tracker 50 processes the image data for the 
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keyframe designated at step S30 to identify further features to be tracked. The 
processing performed at step S3 2 is the same as that performed at step S22 
described above, and consequently will not be described again here. When 
identifying further features to be tracked at step S3 2, however, feature detector 
and tracker 50 discards any feature point which is within a predetermined 
distance of an existing feature point which has been tracked. In this 
embodiment, the predetermined distance is set to five pixels. 

At step S3 4, feature detector and tracker 50 performs tracking of the features 
for the next frame in the sequence. This step is performed for the features 
tracked into the keyframe designated at step S30 (that is, by performing the 
tracking at step S24) and also for the new features identified at step S32. The 
processing performed at step S34 is the same as that performed at step S24, 
described above, and consequendy will not be described again here. 

At step S3 6, keyframe selector 60 determines whether the current frame is 
separated from the preceding keyframe by more than die threshold number of 
frames which was set at step S20. 

If it is determined at step S3 6 that the current frame is not separated from the 
preceding keyframe by more than the threshold number of frames, then the 
processing proceeds to step S48, at which it is determined whether there is 
another frame in the sequence 130. Steps S34, S36 and S48 are then repeated 
imtil it is determined at step S3 6 that the current frame is separated from the 
preceding keyframe by more than the threshold number of frames, at which 
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time the processing proceeds to step S38. 

At step S38, keyframe selector 60 calculates a score representing the total 
relative positional uncertainty of the tracked features in the current frame. 
More particularly, in this embodiment, keyframe selector 60 calculates an 
uncertainty score as follows: 

Relative f ^ 1 

uncertainty = j: min l^.(al.)4 ....(19) 

score nhwest \\d\ 

relative I ' 



uncertainty 
values 



where: 



o, is the measurement variance of the position of the "i"th 

feature point (comer in this embodiment) given by equation (14) 
above; 

di is the disparity vector of the "i"th feature point defining the 
movement of the position of the point from the previous 
keyframe to the current frame; 

a?/|^.|2 is the relative uncertainty value for the "i"th feature 
point; 



(<^^x)' represents an upper threshold on the relative 
uncertainty, and is set to 40 in this embodiment; 
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min(af/|c/ |^(a[;2J^} is the minimum of o'f/\d.\^ and 
(^mL)^ (so that the maximum relative uncertainty value that a 
point can contribute to the summation is (o^^l^;)^ ); and 

the sunmiation is carried out for the lowest "n" relative 
uncertainty values of points which were present in the preceding 
keyframe as well as the current frame (so that d can be 
calculated), "n" being set to 100 in this embodiment. If there 
15 are less than "n" tracked feature points which are present in 

both the current frame and the preceding keyframe, then the 
relative imcertainty score is calculated by adding the values 

20 

min{a^/|6r.|^,(a[^)^} for the "m" existing points which are 
present in both frames and (n-m) maximum x (o^^^J^ (that 
is, (n-m)x40 in this embodiment). 

25 

At step S40, keyframe selector 60 determines whether the relative uncertainty 
score calculated at step S3 8 is less than a threshold value, which, in this 
30 embodiment, is set to 1,000. 

If it is determined at step S40 that the uncertainty value is less than the 
threshold value, then, because the current frame is also separated from the 
preceding keyframe by more than the threshold number of frames (tested at step 
35 S36), at step S42, keyframe selector 60 designates the current frame as a 
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keyframe. 

O. me other hand, if i, is delenmned at step S40 that the relative uncertainty 
score is not less than the threshold value, then processing proceeds to step S48 
so that subsequent frames in d,e input sequence 130 are processed m the 

manner described above. 

At step S44, feamre detector and tracker 50 petfonns processing to refine the 
measurements of the posidons of fl,e Backed features in the 6ame desi^ated 
as a keyframe at Step S42. 



Figure 7 shows operations perfonned by feamre detector and tracker 50 at step 
S44. 



Referring to Figure 7, at step SlOO, featare detector and tracker 50 reads d>e 
position of a,e next tracked feature (flus being the first tracked feahne the first 
<m.e step SlOO is performed) in the new keyframe designated at step S42 and 
the position of the same feature in the preceding keyframe. 

At step S102, feamre detector and Hacker 50 defines a respective window of 
stze 15 pixels by 15 P™ls in the new keyframe and in the preceding keyframe 
centred on die position of the tracked feamre read at step SlOO. 

A. step S104, feamre detector and tracker 50 performs adapdve leas, squares 
.correlationofdiepixelswid^Uiewindowsusingaconventionaltech^^^^^ 
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example as described in "Adaptive Least Squares Correlation: A Powerful 
Image Matching Technique" by A.W. Gruen in Photogrammetry Remote 
Sensing and Cartography, 14(3), 1985 pages 175-187. 

At step S106, feature detector and tracker 50 determines whether the solution 
of the adaptive least squares correlation performed at step S 104 has converged. 

If it is determined at step S 106 that the solution has not converged, then, at step 
S 108, feature detector and tracker 50 discards the position measurement for the 
feature in the new keyframe previously made at step S78 (Figure 6) when 
performing step S34 (Figure 5). In addition, feature detector and tracker 50 
updates the values of the 4D state, X, of the point, the covariance matrix, K, 
and the Kalman filter gain, G, previously calculated at step S82 (Figure 6) when 
performing step S34 (Figure 5) using equations (15), (16) and (17) above and 
die values of X and K existing before the update at step S82 was performed. 

On the other hand, if it is determined at step S106 that the solution has 
converged, then, at step S 1 10, feature detector and tracker 50 updates the values 
of the 4D state, X, of the point, the state covariance matrix, K, and the Kalman 
filter gain, G, previously calculated at step S82 (Figure 6) when performing step 
S34 (Figure 5) using the values of X and K existing before the update at step 
S82 was performed and the value of calculated at step S 104 in equations (9), 
(10) and (11). 

FoUovmig the processing steps SI 00 to SI 10, the current tracked feature has 
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been deal, „iU, and accordmgly. a. step S 1 12. franre detector and tracker 50 
deteonines whether there ,s anodrer .racked featur. in the new keyfran,e and the 
preceding keyfiame. Steps SlOO ,„ S 112 are then repeated undl each tracked 
featnre which is present m bod, the new keyfran,e and the preceding keyftan,e 
has been processed in the manner described above. 

Referring again to Figure 5, at step S46, feature detector and hacker 50 repeats 

a.eprocessmgperfon„edatstepS32f„rthecu.entkeyfran,eto,dendfy further 
feah.es to be hacked. As dus processing has been described above, it will not 
be descnbed again here. 

Ats.epS48.feat,ue detector andtrackerSOdetenuineswhedrerdrereisanoflter 
ftame m dre input sequence 130. and steps S34 to S48 are repeated untU all of 
the frames have been processed m die manner described above. 

Referring to Figures Saand 8b. urwhichFigure 8a showsapart of a.e sequence 
130 of utpu, frames, after perfonmng the processing described above wid, 
respect to Figure 5. processmg apparatus 20 has processed dre image data for 

4emputsequencel30t„designateaplurali^ofd,eframesinfl,e sequence as 

'ceyframes202t„244forsubsequentpr„cessin^Accordi„^y,aseque„ce250 
ofkeyframes has been defmed. in which dre keyframes are arranged in time^f. 
recordutg order and comprise a subset of dre frames m dre origfaal input 

sequence 130. 



Referring again to. Figure 3, at step S6, keyframe selector 



60 together with 
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central controller 30 select so-called "triples" of keyframes, that is respective 
groups each of which comprises three keyframes, for which subsequent 
processing will be carried out to determine transformations defining the 
relationship between the positions and orientations of the camera 140 when the 
keyframes in the group were recorded. 

Figure 9 shows the processing operations performed at step S6. 



ame 



Referring to Figure 9, at step S 132, keyframe selector 60 sets the first keyfr; 
in the sequence 250 (that is, frame 202 in the example shown in Figure 8b) to 
be frame 1 of the first triple. 

At step S 134, keyframe selector 60 considers the next keyframe in the sequence 
250. 

At step S 136, keyframe selector 60 determines whether die number of tracked 
features which are in both the current keyframe considered at step S134 and 
frame 1 of the triple, that is the number of features which have been tracked 
from frame 1 of the triple to the cuixent keyframe, is less than a threshold value, 
which is set to ensure diat frame 1 and frame 3 of a triple are as far apart as 
possible without the number of features fracked between the keyframes falling 
below a threshold. In this embodiment, the threshold is set to 50 features. 

If it is determined at step S136 that the number of tracked features common to 
the current keyframe and frame 1 of the triple is not less than the threshold. 
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steps S134 and S136 are iterativeiy repeated for each subsequent keyfri 
the sequence 250 until it is determined at step S 136 that the number of tracked 
features is less than the threshold value, at which time processing proceeds to 
steps 138. 

At step S138, keyframe selector 60 determines whether there is at least one 
keyframe between die current keyframe and frame 1 of the triple. Referring to 
the example shown in Figure 8b, if keyframe 204 was considered at step S134 
and it was determined at step S 136 that the number of tracked features having 
a measured position in both keyframe 204 and keyframe 202 was less than the 
threshold, dien it would be determined at step S138 that there was not at least 
one keyframe between the current keyframe (frame 204) and frame 1 of the 
triple (frame 202). However, for any other given keyframe in the sequence 250. 
there would be at least one keyframe between the given keyframe and frame 1 
of the triple. 

If it is detennined at step S138 that there is not at least one keyframe in the 
sequence 250 between the current keyframe and frame 1 of the triple, then, at 
step S140. keyframe selector 60 removes frame 1 from die triple and sets the 
current keyframe as frame 1 of a new triple. Thus, referring to die example 
shown in Figure 8b, keyframe 202 would no longer be designated as frame 1 of 
the first triple, and instead keyframe 204 would be designated as frame 1. 
Processing dien returns to step S134 to consider the next keyframe in die 
sequence 250, as described above. 
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On the other hand, if it is determined at step S138 that there is at least one 
keyframe in the sequence 250 between the current keyframe and frame 1 of the 
triple, then, at step S 142, keyframe selector 60 sets the keyframe preceding the 
current keyframe in the sequence 250 as frame 3 of the triple. Thus, referring 
to the example shown in Figure 8b, if the current keyframe being processed is 
keyframe 218 and it is determined at step S136 that the number of tracked 
features is less than the threshold then, because there is at least one keyframe 
between keyframe 218 and keyframe 202 (which is frame 1 of the triple), at 
step S142, keyframe 216, (that is, the keyframe immediately preceding 
keyframe 218 in the sequence 250), is set as frame 3 of the first triple. 

By performing processing to set frame 3 of the triple as described above, it is 
ensured that frames 1 and 3 of the triple are as far apart as possible in the 
sequence 250 (and hence in distance aroxmd object 120, too) without the 
number of tracked features which are common to the keyframes falling below 
a threshold. This provides a wide baseline between frame 1 and frame 3 of the 
triple, which enables more accurate camera transformations to be calculated 
since, in general, the wider the baseline, the more accurate the calculated 
camera transformations. 

At step S144, keyframe selector 60 determines whether there is at least one 
keyframe between frame 1 of the triple and frame 3 of the triple which w£is set 
at step S142. 

If it is determined at step S 144 that there are no keyframes between frame 1 and 
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frame 3 of the triple, then processing proceeds to step S146, at which frame 3 
IS removed from the triple (and consequently becomes a nomal keyframe 
again), and then to step S140, at which frame 1 is removed from the triple and 
the current keyframe is set as frame 1 of a new triple. The processing then 
returns to step S134 to consider the next keyframe in the sequence 250 as 
described above. 

On the other hand, if it is determined at step S144 that there is at least one 
keyframe between frame 1 and frame 3 of the triple, then, at step S148. 
keyframe selector 60 determines whether there is an odd number of keyframes 
between frame 1 and frame 3 of the triple. 

If it is determined at step 8148 that there is an odd number of frames between 
frame 1 and frame 3 of the triple, then, at step S150, keyframe selector 60 sets 
the middle frame of the keyframes between frame 1 and frame 3 as frame 2 of 
the triple. 

On the other hand, if it is determined at step S148 that there is an even number 
of keyframes between frame 1 and frame 3 of the triple, then, at step .8152, 
keyframe selector 60 sets frame 2 of the triple to be the frame from the middle 
pair of keyframes which contains the highest number of tracked features which 
are also present in frame 1 and frame 3 of the triple. Thus, referring to Figure 
8b by way of example, there is an evennumber of keyframes between keyframe 
202 (frame 1 of a triple) and keyframe 216 (frame 3 of a triple). Accordingly, 
keyframe selector 60 selects either keyframe 208: or keyframe 210 as frame 2 
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of the triple (keyframe 2 10 in the example shown in Figure 8b) depending upon 
which of these two keyframes has the highest number of tracked features which 
are also present in keyframe 202 and keyframe 216. 

5 At step S154, having set up a triple of keyframes, keyframe selector 60 
considers the next keyframe in the sequence 250 after frame 3 of the preceding 
triple (keyframe 218 in the example shown in Figure 8b). 

At step S 156, keyframe selector 60 determines whether the number of tracked 
features which appear in both the current keyframe considered at step S 154 and 
frame 3 of the preceding triple is less than a threshold. This threshold is set to 
a low value to ensure that the new triple of keyframes to be created is 
sufficiently independent from the preceding triple of keyframes (that is, the 
views of the object 120 present in the new triple to be set up are sufficiently 
different from the views of the keyframes in the preceding triple). In this 
embodiment, the threshold value is set to 10 features. 

If it is determined at step S156 that the number of tracked features is not less 
than the threshold, then processing retums to step S154 to consider the next 
20 keyframe in the sequence 250. Steps S 154 and S 156 are repeated to consider 
each fiirther keyframe in the sequence 250 in turn imtil it is determined at step 
S 156 that the number of tracked features which are in both the keyframe being 
considered and frame 3 of the preceding triple is less than the threshold. 

25 When it is determined at step S156 that the number of tracked features in the 
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current keyframe and frame 3 of the preceding triple is less than the threshold, 
then, at step S158, keyframe selector 60 sets the current keyframe as frame 1 
of a new triple. 

Processing then returns to step S 134 to defme frame 3 and frame 2 of the triple 
and to defme further friples of keyframes in the manner described above. 

After performing the processing described above with respect to Figure 9, 
processing apparatus 20 has processed the keyframes in sequence 250 to defme 
a plurality of triples of keyframes, each triple comprising three of the keyframes 
from the sequence 250. In addition, as a result of selecting the keyframes to 
make up a triple as described above, none of the resulting triples overlap in the 
sequence 250. 

As shown in the example in Figure 8b. the keyframes within a triple will 
typicaUy have fiirther keyframes therebetween (for example keyframes 204, 206 
and 208 are between frame 1 and frame 2 of the first triple and keyframes 212 
and 214 are between frame 2 and frame 3 of the first triple). . Further, 
successive triples Avill also typicaUy have keyframes therebetween (for example 
keyframes 218 to 224 are between the first triple and the second triple in the 
example shown in Figure 8b). 

Referring again to Figure 3, at step S8, camera transformation calculator 70 
calculates and tests transformations defming the relationships between the 
positionsand orientations at which frame l,. frame ,2 and frame 3 of each of the 
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triples of keyframes defined at step S6 were recorded. The calculated 
transformations are then stored in camera transformation store 100. 

In this embodiment, for each given triple of keyframes, camera transformation 
calculator 70 calculates transformations for frame 2 and frame 3 which are 
relative to frame 1 of the triple, that is, which define the movement of the 
camera 140 to frame 2 and frame 3 from the position of frame 1 in the given 
triple. As a result of this processing, a respective "set" of keyframes is created 
from each triple, a set being a group of keyframes in which the camera 
projection for each frame in the set is defined relative to another frame in the 
set. 

Figure 10 shows the processing operations performed by camera transformation 
calculator 70 at step S8. 

Referring to Figure 10, at step S 180, camera transformation calculator 70 reads 
data previously generated by feature detector and tracker 50 defining the 
positions of matching feature points (points which represent the same actual 
point on the object 120) in frame 1, frame 2 and frame 3 of the next triple (this 
being the first triple the first time step SI 80 is performed). That is, camera 
transformation calculator 70 reads data defining the positions of feature points 
tracked between frame 1 and frame 2 of the triple, between frame 1 and frame 3 
of the triple, and between frame 2 and frame 3 of the triple. 

At step S182, camera transformation calculator 70 calculates and tests 
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transforations which define the movement of the camera 140 from the 
position and orientation at which frame 1 of the triple was recorded to the 
position and orientation at which frame 2 of the triple was recorded, and 
ti-ansformations which define the movemem of the camera. 140 from the 
position and orientation at which frame 2 of the triple was recorded to the 
position and orientation at which frame 3 of the triple was recorded. 

To perform the processing at step S182, camera transformation calculator 70 
uses a first procedure defined in procedure instiuction store 80. More 
particularly, in this embodiment, the processing perfonned at step S182 by 
camera transformation calculator 70 is the same as the processing described 

witiirespecttoFigures21to35inEP-A-0898245.EP-A-0901105,andpending 
US appUcations 09/129077, 09/129079 and 09/129080. The M contents of 

pending US appUcations 09/129077, 09/129079 and 09/129080 are incorporated 
herein by cross-reference. Accordingly, the processing will not be described 
again here in detail. In summary, however, to calculate the relative camera 
transformations at step S 182, camera transformation calculator 70 calculates a 
m perspective transformation and a scaled affine transformation defining the 
movement between frame 1 and frame 2 in the triple, and a Ml perspective 
ti-ansformation and a scaled affine transfonnation defining Ae movement 
between frame 2 and frame 3 in the triple. Camera transformation calculator 
70 then considers each combination of these transformations to select the best 
transformations defining the movement between frame 1, frame 2 and frame 3 
as a whole. To test the calculated transformations for die triple as a whole, 
cameratransformatiohtilculatof 70usesmatctiin^fea i J 
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frame 3 of the triple and the calculated positions and orientations of these 
frames to estimate the position in three-dimensions of the point on the object 
120 which the feature points represent. The estimated 3D point is then 
projected into frame 2 of the triple using the calculated transformations, and the 
5 distance between the projected point in frame 2 and the position of the feature 
point in frame 2 as measured by feature detector and tracker 50 is calculated. 
The total number of points for which this distance is less than a threshold is 
then used as an accuracy measure of the calculated transformations, together 
with an error measxu-ement based on the total of all of the distances between 
10 projected points and measured points in frame 2. 

At step SI 84, camera transformation calculator 70 determines whether the 
transformations calculated at step SI 82 are sufficiendy accurate. The 
processing performed at this step comprises reading whether the tests performed 
15 at step SI 84 determined whether the calculated transformations were 
sufficiently accurate (determined at step S362 or S370 in Figure 28 of EP-A- 
0898245, EP-A-0901105, or pending US appUcations 09/129077, 09/129079 
and 09/129080). 

20 If it is determined at step SI 84 that the calculated transformations are not 
sufficiently accurate, then, at step SI 86, camera transformation calculator 70 
dissociates the triple of keyframes currently being considered. On the other 
hand, if it is determined at step SI 84 that the calculated transformations are 
sufficiently accurate, then step SI 86 is omitted, and the calculated 

25 transformations are stored in camera transformation store 100. 
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At step S 188, camera transfonnation calculator 70 determmes whether there is 
another triple of keyframes to be processed. Steps S180 to S188 are repeated 
until each tnple of keyframes has been processed in the mamier described 

above. 



After performing theprocessing at stepSS, camera transformationcalculator20 
has formed a set of keyframes from each triple of keyframes by calculating, for 
each triple, the position and orientation of frame 2 in the triple relative to frame 
1 in the triple and the position and orientation of frame 3 in the triple relative 
to frame 2 in the triple. Thus, referring to Figure 11, the position and 
orientation of keyframe 210 has been calculated relative to keyframe 202, and 
the position and orientation of keyframe 216 has been calculated relative to 
keyframe 210. Similarly, the position and orientation of keyframe 232 has been 
calculated relative to keyframe 226, and the position and orientation of 
keyframe 236 has been calculated relative to keyframe 232. 

At this stage, transformations have not been calculated for keyframes between 
the keyframes making up a triple (that is, the frames 204, 206. 208, 212. 214. 
228, 230, 234 and 244 in the example shown in Figure 1 1) or for keyframei 
between frame 3 of one triple and frame 1 of the next triple (that is, keyframes 
218, 220, 222, 224, 238 and 240). 



Referring again to Figure 3, at step SIO, camera transfonnation calculator 70 
considers each triple of keyframes which has additional keyframes between 
frame 1 and frame 2 and/or between frame 2 and frame 3 of the triple, and 
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performs processing to calculate camera transformations for the additional 
keyframes. 

More particularly, at step SIO, camera transformation calculator 70 calculates 
relative transformations for each keyframe lying between frame 1 and frame 2 
and between frame 2 and frame 3 of the triple (that is, transformations defining 
the movement of the camera 140 from frame 1 of the triple to the keyframe), 
and subsequently refines the calculated camera transformations. 

Figure 12 shows the processing operations performed by camera transformation 
calculator 70 at step SIO. 

Referring to Figure 12, at step S200, camera transformation calculator 70 reads 
data defining which keyframes make up the next remaining triple of keyframes 
(that is, the next triple which remains after the processing described abave with 
respect to Figure 10 has been performed, in which some triples may have been 
dissociated if the calculated camera transformations were not sxxfSciently 
accurate). 

At step S202, camera transformation calculator 70 determines whether there are 
any keyframes between frame 1 and frame 2 of the triple and/or between 
frame 2 and frame 3 of the triple. If it is determined that there are no such 
keyframes, then processing proceeds to step S208, at which it is determined 
whether there is another remaining triple of keyframes, and, if there is, to step 
S200 at which the next triple is considered. 
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On the other hand, if it is determined at step S202 that there is at least one 
keyframe between frames 1, 2 and 3 of the triple, then, at step S204, camera 
transformation calculator 70 calculates relative camera transformations for each 
of the keyframes lying between the keyframes of the triple. 

Figure 13 shows the processing operations performed at step S204 by camera 
transformation calculator 70. 

Referring to Figure 13, at step S220, camera transformation calculator 70 
calculates the position in three-dimensions of a feature point for each feature 
tracked by feature detector and tracker 50 which is present in at least two of the 
three keyframes of the triple. Each feature point calculated at step S220 
therefore represents the position of an actual point on the object 120 in three- 
dimensions predicted on the basis of the position of the feature in the keyframes 

and thepositions and orientationsofthekeyframes previously calculated at step 
S8. In this embodiment, camera transformation calculator 70 calculates the 3D 
position of each feature point using the technique described in EP-A-0898245. 
EP-A-0901105 and pending US applications 09/129077, 09/129079 and 
09/129080 with respect to Figures 4 1 to 48 therein. 

At step S222, camera transformation calculator 70 reads data for the next 
keyframe (this being the first keyframe the first time step S222 is performed) 
which is between the keyframes of the triple (an "intra-triple" keyframe). 

At step S224, camera transfoimation calculator' 70 generates ' a list of 
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correspondences between the 3D positions of the feature points calculated at 
step S220 and the 2D positions of the corresponding features in the intra-triple 
keyframe for which data was read at step S222. More particularly, referring to 
Figure 14, for each feature point 1 to "n" which has both a 2D position in the 
5 intra-triple keyframe (previously calculated by feature detector and tracker 50) 
and a 3D position (calculated at step S220), camera transformation calculator 
70 lists the coordinates 260 of the 3D position and the coordinates 262 of the 
2D position. 

10 At step S226, camera transformation calculator 70 increments the value of a 
counter by 1 (the counter being set to the value 1 the first time step S226 is 
performed). 

At step S228, camera transformation calculator 70 selects at random three of the 
15 correspondences listed at step S224 (each correspondence defining the 
coordinates 260 of the feature point in 3D and the coordinates 262 of the 
feature in the intra-triple keyframe). 

At step S230, camera transformation calculator 70 uses the three 
20 correspondences selected at step S228 to calculate transformations defining the 
movement of the camera 140 from frame 1 in the triple to the intra-triple 
keyframe, in accordance with instructions stored in procedure instruction store 
80 defining a second transformation calculation procedure. More particularly, 
camera transformation calculator 70 performs processing at step S230 to 
25 . calculate the transformations in a conventional manner, for example as 
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described in section 13.4.15 of "Computer and Robot Vision Volume 2", by 
R.M. HaralickandL.G. Shapiro, Addison-Wesley Publishing Company, 1993, 
ISBN 0-201-56943-4 (v.2). 

At step S232, camera transformation calculator 70 tests the transformations 
calculated at step S230 against each of the correspondences which were listed 
at step S224. More particularly, for each feature point listed in the 
correspondences generated at step S224, camera transformation calculator 70 
calculates the distance between the position of the feature in the intra-triple 
keyframe as measured by the feature detector and tracker 50 (defined by the 
U,V coordinates 262 in Figure 14) and the position in the intra-triple keyframe 
of the corresponding 3D feature point calculated at step S220 when projected 
mto the intra-triple keyframe using the transformations calculated at step S230, 
and determines whether the calculated distance between these positions is less 
than a threshold. This processing is performed using the foUowing equation, 
and is repeated for each point listed in the correspondences generated at step 
S224, to determine the number of points for which the inequahty holds: 



PK. 
X - L 



....(20) 

where: X; is the position of the "i"th tracked feature in the intra-triple 
keyframe, given by Xi = (u;, v;, 1); 

2Ci is the position of the "i"th tracked feature in 3D (calculated at 
■ ' step S220),'giveh by Xi= (x; yi, zTVjy *■ ' : : 
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"P" defines the camera transformations calculated at step S230, 
which is a 3 by 4 matrix, and is given in a conventional manner 
by P = K [R t], which K is a matrix defining the intrinsic 
parameters of the camera 140, "R" is the calculated rotation and 
5 "t" is the calculated translation; 

(PXj)3 is the third component of the vector PX^; 

P2Ci/(PXi)3 is the predicted position of the "i"th 3D feature point 
10 in the intra-triple keyframe on the basis of the calculated camera 

transformations; and 

"d" is a threshold value, which, in this embodiment, is set to 
d = 3 pixels. 

At step S234, camera transformation calculator 70 determines whether the 
camera transformations calculated at step S230 and tested at step S232 are more 
accurate than any camera transformations calculated when step S230 was 
performed during a previous iteration. More particularly, camera 
transformation calculator 70 determines from the result of the tests performed 
at step S232 whether the number of points which satisfy the inequality given in 
equation (20) is greater than the number of points which satisfied the inequality 
when the current most accurate camera transformations were tested. 
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If it is determined at step S234 that the calculated transformations are more 
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accurate than any calculated at step S230 during a previous iteration, then, at 
step S236, camera transformation calculator 70 stores the calculated 
transformations together with the number of points satisfying the inequahty 
defined in equation (20) in camera transformation store 100. On the other hand, 
if it is determined that step S234 that the camera transformations are not more 
accurate than any previously calculated, then step S236 is omitted. 

AtstepS238, camera transformation calculator 70determineswhetherthevalue 
of the comiter incremented at step S226 is less than a threshold number defming 
the maximum number of iterations of steps S226 to S236 to be performed, 
which, in this embodiment, is set to 100,000. 

If it is determined at step S238 that die counter value is less than the threshold 
(indicating that the required nmnber of iterations have not yet been performed), 
then, at step S240, camera transformation calculator 70 detennines whether the 
accuracy of the camera transformations calculated at step S230 (represented by 
the nmnber of points satisfying the inequality in equation (20) tested at step 
S232) has increased in the last 20,000 iterations of steps S226 to S238. 

If it is determined at step S240 that the accuracy has increased during die last 
20,000 iterations, then it is determined that it is worthwhile performing further 
iterations, and steps S226 to S240 are repeated. 

On die other hand. .if it is determined at step S240 diat tiiere has not been any 
change in the accuracy -of the calculated camera transformations: or^if it is 
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determined at step S238 that the counter value is no longer less than the 
threshold, then no further camera transformations are calculated for the current 
intra-triple keyframe, and processing proceeds to step S242. 

At step S242, camera transformation calculator 70 determines whether the 
accuracy of the camera transformations for the intra-triple keyframe currently 
stored (that is, the most accurate transforaiations calculated) exceeds a 
predetermined threshold. More particularly, camera transformation calculator 
70 determines whether the number of points satisfying the inequality given by 
equation (20) for the stored camera transformations exceeds the value five. 

If it is determined at step S242 that the accuracy of the camera transformations 
does not exceed the threshold, then, at step S244, the intra-triple keyframe is 
discarded so that it no longer forms part of the sequence of keyframes 250. On 
the other hand, if it is determined at step S242 that the accuracy of the stored 
transformations does exceed the threshold, then step S244 is omitted. 

At step S246, camera transformation calculator 70 determines whether there is 
another keyframe which is between the keyframes of the triple currently being 
considered, and steps S222 to S246 are repeated until each intra-triple keyframe 
has been processed in the manner described above. 

As a result of performing the processing at step S204 (Figure 12) to calculate 
camera transformations defining the position and orientation of each intra-triple 
keyframe within a given triple, processing apparatus 20 has increased the 
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number of keyframes in the set (which originally comprised just the triple of 
keyframes). The position and orientation of each keyframe in the set is defined 
relative to the fu-st frame in the set (that is, frame 1 of die original triple). Thus, 
referring to the example shown in Figure 15a, an expanded set 266 of 
keyframes is created by performing the processing described above with respect 
to Figure 13 for the first triple of keyframes in the example shown in Figure 1 1. 
The expanded set 266 comprises the keyframes 202, 2 10 and 2 16 making up the 
original triple together with the original intra-triple keyframes 204, 206, 208, 
212 and 214. As discussed previously, the position and orientation of each of 
keyframes 204 to 2 16 is defined relative to keyframe 202. 

Referring again to Figure 12, at step S206, camera transformation calculator 7a 
uses a third transformation calculation procedure defined by instructions stored 
in procedure instruction store 80 to refine the camera transformations for the 

expandedsetofkeyframes generated at stepS204.so that error in thecalculated 
transformations is minimised and distributed evenly through the keyframes in 
the set. The refined transformations are then stored in camera transformation 
store 100. 



camera 



Referring to Figures 15a to 15d, in this embodiment, at step S206, 
transformation calculator 70 refines the camera transfonnations in stages, 
considerihg at each stage the calculated camera transformations for a different 
subset of the keyframes within the overaU set. More particularly, referring to 
Figure 15a, in the first stage of processing, camera transformation calculator 70 
considers a:"wind.ow" 270 containing a number, of keyframes, and refines the 
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camera transformations for the keyframes within the window. Refening to 
Figure 15b, during the second stage of processing, camera transformation 
calculator 70 moves die window 270 by one keyframe within the set 266, and 
considers the next subset of keyframes which now he within the window to 
refine the camera transformations therefor. Similarly, referring to Figure 15c 
and Figure 15d, during subsequent stages of processing, camera transformation 
calculator 70 again moves the window 270 by one keyframe at each stage and 
refines the calculated camera transformations for the keyframes within the 
window. In this way, the camera transformations for every keyframe within the 
set 266 are refined, but only a subset (window) of the keyframes are considered 
at each stage, thereby significantly reducing processing requirements. This 
method of refining the camera transformations is particularly applicable where 
there are a large number of keyframes and/or a large number of tracked features 
in the keyframes (rather than the small number of keyframes within set 266 
shown in Figures 15a to 15d, which is used for illustration purposes only). 

Figure 16 shows the processing operations performed by camera transformation 
calculator 70 at step S206. 

Referring to Figure 16, at step S270, camera transformation calculator 70 reads 
the 2D position of each tracked feature point in each of the keyframes within 
the next window (this being the first window, that is the first selected nvunber 
of keyframes in the set, the first time step S270 is performed). 

In the example shown in Figures 15a to 15d, the window 270 has a size of five 
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keyframes. However, in practice, the size of the window 270 is set taking into 
account the number of keyframes within the set 266 to be processed, the 
number of tracked features therein, the time available to carry out the 
processing, and the processing resources available to carry out the processing. 
More particularly, as camera transformation calculator 70 increases the number 
of keyframes within the window 270, die amount of processing operations to 
be performed increases, although the accuracy of the adjusted camera 
transformations will also increase. 

Referring to Figure 17, an example is shown of a list of each tracked feature 
point 1 to "n" which appears in any of the keyframes in the set 266 in the 
example shown in Figures 15a to 15d, together with tiie position (U. V) of each 
feature point in each keyframe within die set 266. For example, tracked feature 
point 1 has a position (U„, V„) within frame 1 (diat is, frame 202 of the set), 
a position (U^,, V^,) in the second keyframe 204 of the set, a position (U,i, V3,) 
in the third keyframe 206 of die set, etc. Similarly, tracked feature point 2 im 
a position (U.„ V,,) in the first keyframe 202 of the set etc. but was not tracked 
into die fourth to eighdi frames in the set by feature detector and tracker 50 and 
accordingly does not have a position widiin these frames. 

At step S270, dierefore, camera transformation calculator 70 reads the 2D 
coordinates (U, V) widiin die window 270 shown in Figure 17, diat is die 
coordinates of each tracked feature point which has a position widiin at least 
one keyframe widiin die window 270. Thus, refening to die example shown 
in Figure 17, die coordinates widiin die window 270 of each of die tracked 
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feature points 1, 2, 3, 4 and "n" (and possibly others between 5 and "n") is read 
since each of these points has a (U, V) value (that is, a measured position) 
within the window 270. However, the tracked feature point 5 is not taken into 
consideration because this point does not have a position in any of the 
keyframes within the window 270. 

Referring again to Figure 16, at step S272, camera transformation calculator 70 
calculates an error measurement value using the 2D positions of the tracked 
features read at step S270, the 3D positions calculated at step S220 (Figure 13) 
which correspond to the tracked features read at step S270, and the camera 
transformations for the subset of keyframes within the window 270 previously 
calculated at step S182 (if the keyframe was part of the original triple of 
keyframes) or at step S204 (if the keyframe was an intra-triple keyframe). 

More particularly, in this embodiment, camera transformation calculator 70 
calculates the error measxirement value by deterniining, for each keyframe in the 
window 270 and each tracked feature point which has a measured position in 
at least one of the keyframes in the window 270, the difference between the 
position of the tracked feature point in the keyframe as measured by feature 
detector and tracker 50 and the position of the corresponding 3D point when 
projected into the keyframe using the calculated camera transformations, 
squaring these distances and summing the squared distances: More particularly, 
camera transformation calculator 70 calculates an error measurement value, 
E(P,X), as follows: 



54 263590\ 



5 where: "i" labels keyframes within the window 270 (in the example 
shown in Figure 17, i=1...5); 

"j" labels the tracked feature points which have a position 
measured by feature detector and tracker 50 in at least one of die 
keyframes in the window 270; 

"a" = 1, 2, being the column and die row of the position of a 
tracked feature point or projected point in a keyframe; 
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"x" is the position of a tracked feature point in a keyframe within 
window 270 measured by feature detector and tracker 50, given 
in terms of the column or row (depending on die value of "a") of 
the pixel representing die tracked feature point (diis may be a 
subpixel value, that is, the column or row values are not 
necessarily integers); 

"X" is die position of a feature point in 3D (calculated at step 
S220), given by X = (x, y, z, 1); 

"P" defines die camera transfomations for a keyframe (calculated 
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at step S 182 or step S204). "P" is a 3 by 4 matrix, and is given 
in a conventional manner by P = K[R t], in which "K" is a matrix 
defining the intrinsic parameters of the camera 140, "R" is the 
calculated rotation and "t" is the calculated translation. The 
camera transformations for the first keyframe in a set (frame 202 
in the example shown in Fig. 15a) are given by P = K[I, O], 
where "I" is the identity matrix (since the transformation for 
subsequent keyfirames are defined relative to the first keyfi-ame); 

(PX)3 is the third component of the vector PX; 

PX/(PX)3 is the predicted position when a 3D feature point is 
projected into a keyfi-ame on the basis of the calculated camera 
transformations. The predicted position is given in terms of the 
column or row (depending on the value of "a") of the pixel 
representing the projected feature point; 

"W" is a weighting factor, which, in this embodiment, is fixed 
and is set to: 

fF.. = -L ....(22) 

where represents the positional uncertainty, and is set, in this 
embodiment, to = 1 . 
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At step S274, camera transformation calculator 70 adjusts the previously 
calculated camera transformations of the keyframes inthe window 270 (that is, 
P in equation 21) and the 3D positions of selected feature points (that is, "X" 
in equation 21) previously calculated at step S220 in accordance with a 
minimisation algorithm to minimise the error measuremem value calculated at 
step S272. More particularly, camera transformation calculator 70 adjusts the 

calculated camera transformations (P) for each keyframe in the wmdow 270 and 
the 3D positions (X) for each feature point which corresponds to a tracked 
feature having a measured position (u, v) withm at least one keyframe in the 
window 270 but no measured position in keyframes within the set which are 
outside the window 270. Thus, referring to the example shown in Figure 17, 
the camera transformations for the keyframes 1 to 5 and the 3D positions of 
feature point 2 and feature point 3 are adjusted, but not the 3D positions of 
feature points l,4and V. After adjusting the camera transformations (P) and 
the 3D positions (X), camera transforation calculator 70 performs processing 
to recalculate the error measurement value in accordance with equation (2 1) by 
using the adjusted camera transformations to reproject the adjusted 3D feature 
points and the michanged 3D feature points which correspond to a tracked 

featm-ehavingameasuredposition(u,v)inatleastonekeyframein the window 
270 and a measured position (u, v) in another keyframe in the set outside the 

window 270. 

This processing is performed in an iterative mamier. with the camera 
transformations (P) and 3D positions (X) being adjusted and the error 
measurement value being recalculated in accordance witf, equation (21) using 
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the adjusted values at each iteration, and the iterations being continued until the 
error measurement value converges to a minimum value. In this embodiment, 
when adjusting the camera transformations, P, the values defining the intrinsic 
parameters, K, of the camera 140 remain fixed, and the camera transformations 
for the keyframe in the set which was originally frame 1 of the triple also 
remain fixed (since the camera transformations for all of the other keyframes 
in the set are defined relative to frame 1 of the triple). 

More particularly, in this embodiment, camera transformation calculator 70 
performs processing in accordance with a Levenberg-Marquardt minimisation 
algorithm based on the method described in chapter 15.5 of "Numerical Recipes 
in C" by Press et al, Cambridge University Press, ISBN 0-521-43108-5 (in 
which x^(a) corresponds to E(P,X) in equation (21) above). However, different 
criteria are used to determine whether to increase or decrease A. More 
particularly: 

if x^(a+6a)^x^(^)"^5 then X is increased by a factor of 10; or ...(23) 

if x\a+6a)<x^(a)-€, then A is decreased by a factor of 10. ....(24) 

where e = 3 x y/2n ....(25) 

where "n" is the number of terms (labelled by i, j, a) in the simulation of 
equation (21) above. 
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Since the value of e depends on the standard deviation of the r distribution, 
equations (23) and (24) ensure that statistically insignificant changes in the 
value of are ignored. In addition, equations (23) and (24) enable criteria 
based on the value of X alone to be used in this embodiment to determine when 
to stop iterating. More particularly, iterations are stopped when: 



^-^^ ....(26) 



where is a threshold constant, which, in this embodiment, is set to Ixio*. 

At step S276, camera transformation calculator 70 detennines whether there is 

another window ofkeyframes in the set ofkeyframes currently beingprocessed. 
More particularly, camera transformation calculator 70 determines whether the 
last keyframe within tiie window for which calculations have just been 
performed is the last keyframe within the set. If it is determined that the last 
keyframe in the window is not the last keyframe in the set (for example as 
shown in Figure 15a since keyframe 208 is not the last keyframe in the set 266) 
then, processing returns to step 8270, at which the next window of keyframes 

is considered(as noted above, this being the window displaced byonekeyframe 
with respect to the previous window so that the first keyframe in the previous 
window drops out of the window). 

Steps S270 to S276 are repeated mitil each window of keyframes in the set has 
been processed in the mamier described above (that is, when it is determined at 
step S276 that the last keyframe in the window is the last keyframe in the set, 
for example as shown in Figure 15d). 
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Referring again to Figure 12, at step S208, camera transformation calculator 70 
determines whether there is another remaining triple of keyframe in the 
sequence 250. Steps S200 to S208 are repeated until each triple of keyframes 
has been processed in the maimer described above. 

Referring to Figure 18, as a result of performing the processing at step SIO 
(Figure 3), processing apparatus 20 has generated a plurality of expanded sets 
266, 276, 286 of keyframes within the sequence 250. Within each set, the 
position and orientation of each keyframe has been calculated relative to the 
first keyframe in the set (keyframe 202 in set 266, keyframe 226 in set 276 and 
keyframe 242 in set 286). Any keyframe which was discarded at step S244 
(Figure 13) because the calculated camera transformations were not siifficiently 
accurate will leave a gap in the sequence 250, such as the gap between 
keyframes 228 and 232 in set 276. 

At this stage, the positions and orientations of the keyframes between sets 
(keyframes 218, 220, 222, 224, 238 and 240 in the example shown in Figure 
1 8) remain uncalculated. 

Referring again to Figure 3, at step S12, camera transformation calculator 70 
performs processing to join the sets of keyframes within the sequence 250, and 
stores the results in camera transformation store 100. 

Figure 19 shows the processing operations performed by camera transformation 
calculator 70 at step SI 2. 
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Referring to Figure 19. at step S290, camera transformation calculator 70 
calculates transformations for keyframes within sequence 250 which are 
between sets of keyframes. This is performed by extending the sets of 
keyframes to further expand the number of keyframes in the sets, as will now 
be described. 



Figure 20 shows the processing operations perfonned by camera transfomation 
calculator at step S290. 

Referring to Figure 20, at step S300, camera transformation calculator 70 
defines an order in which the ends of the sets of keyframes are to be processed 
to extend the sets. More particularly, in this embodiment, camera 
transformation calculator 70 defines a processing order based on the accuracy 
of the keyframe at the beginning and end of each set (diat is, keyframes 202 and 
216.226and236and242intheexampleshowninFigure 18) so that these end 
keyframes are processed in the order of decreasing accuracy. The accuracy 
used by camera transformation calculator 70 to define the processing order at 
step S300 is the accuracy calculated at step S182 (Figure 10). TWs accuracy 
represents the accuracy of the keyframes in the original triple, and accordingly 
the first keyframe and the last keyframe in each respective set will have the 
same accuracy values. For this reason, in this embodiment, frame 1 in a set is 
placed in the processing order so that it is considered before frame 3 (the end 
keyframe) in a set. 

At step S302, camera transformation calculator 70 considers the next end 
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keyframe in the processing order defined at step S3 00 (that is, the end keyframe 
of highest accuracy), and, at step S304, considers the keyframe adjacent to the 
end keyframe in the gap between the sets. Thus, referring to Figure 18 by way 
of example, if the processing order defines keyframe 226 as the next keyframe 
to be considered, then, at step S3 04, keyframe 224 is considered. Similarly, if 
the processing order defines keyframe 216 as the next keyframe to be 
considered, then, at step S3 04, keyframe 218 is considered. 

Referring again to Figure 20, at step S306, camera transformation calculator 70 
reads which tracked features are present in the keyframe considered at step 
S304 and identifies the corresponding 3D points which were calculated at step 
S220 (Figxire 13) or at step S274 (Figure 16). If no 3D point exists for a tracked 
feature point which has a measured position in the keyframe, and the tracked 
featvu-e point is also present in at least two keyfi:ames of the set which contains 
the end keyframe considered at step S3 02, then camera transformation 
calculator 70 calculates a 3D point in the maimer described previously at step 
S220. 

At step S308, camera transformation calculator 70 calculates relative camera 
transformations for the gap keyframe, that is transformations defining the 
rotation and translation of the keyframe relative to frame 1 in the set which 
contains the end keyframe considered at step S302. The processing performed 
by camera transformation calculator 70 at step S3 08 is the same as that 
performed at step S204, which is described above with respect to Figure 13, and 
accordingly will not be described again here. 
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At step S3 10, camera transformation calculator 70 tests whether the 
transformations calculated at step §308 are sufficiently accurate (this step 
corresponding to step S242 in Figure 13). 

If it is determined at step S3 10 that the camera transformations are not 
sufficiently accurate, then, at step S3 12, camera transfoimation calculator 70 
discards the transfomiations calculated at step S308. and deletes the keyfi-ame 
considered at step S302 fi-om the processing order (so diat it is not considered 

again). 

On the other hand, if it is determined at step S3 10 that the camera 
transformations calculated at step S308 are sufficiently accurate, the 
transformations are retained so diat the gap keyframe becomes part of the set 
containing the end keyframe considered at step S302. Then, at step S314, 
camera transformation calculator 70 adjusts the calculated camera 
transfomations for the extended set including the keyframe which was 
previously in the gap, so that error in the calculated transformations is 
minimised and distributed evenly through a number of the keyframes in the 
extended set. More particularly, camera transformation calculator 70 peiforms 
the same processing as the processing at steps S270, S272 and S274 (Figure 
16), with the window 270 in a fixed (non-moving) position such that the gap 
keyframe for which camera transformations were calculated at step S308 is the 
first keyframe in the window if the gap keyframe was before the set in the 
sequence 250 or the last keyframe in the window if the gap keyframe was after 
the set in the sequence 250. As .this processing has been, described above, it 
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will not be described again here. 

At step S3 16, camera transformation calculator 70 updates the processing order 
for the ends of the sets of keyframes to take into account the keyframe 
considered at step S304, which now forms the end of a set. More particularly, 
camera transformation calculator 70 deletes the old end keyframe considered 
at step S302 from the processing order (so that it is not considered again) and 
inserts the new end keyframe in the processing order in accordance with the 
accuracy of the calculated transformations read at step S3 10. 

At step S3 1 8, camera transformation calculator 70 determines whether there is 
another end keyframe in the processing order list. Steps S3 02 to S3 18 are 
repeated until there are no longer any end keyframes in the processing order list, 
that is, until each keyframe set has been expanded to the point where it is no 
longer possible to expand the sets without the camera transformations falling 
below an acceptable accuracy. 

By performing the processing at step S290 (Figure 19) in the manner described 
above with respect to Figure 20, processing apparatus 20 attempts to extend the 
sets of keyframes such that the most accurate end of a set is always considered 
for extension before a lower accuracy end. This results in higher accuracy 
camera transfonnations being calculated for the keyframes within the gaps 
between sets. Further, by expanding sets of keyframes in both directions within 
the sequence 250 (that is, from the first keyframe in the set and the last 
keyframe in the set) if camera transformations for a keyframe within a gap 
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(such as keyframe 240 in Figure 18) can not be calculated with sufficient 
accuracy relative to the set on one side (such as set 286) then it may still be 

possibleto calculate sufficientlyaccurate camera transfonnations relative to the 
set on the other side (set 276) when this set is expanded towards the keyframe 



in question. 



As a result of the processing at step S290, some of the sets of keyframes will 
have been expanded to such an extent that there is no longer a keyframe 
separating adjacent sets in the sequence, while in odier cases, adjacent sets in 
the sequence will remain separated by at least one keyframe. Referring to 
Figure 21 by way of example, set 266 has been expanded to include keyframe 
218 at its end. and set 276 has been expanded to include keyframe 224 at its 
start and keyframes 238 and 240 at its end. Thus, sets 276 and 286 now meet 
(they are not separated by any keyframe in the sequence 250) while sets 266 
and 276 have a gap therebetween as they are still separated by keyframes 220 



and 222 

Referring again to Figure 1 9. at step S292. camera transformation calculator 70 
merges sets of keyframes which meet in the sequence 250 (sets 276 and 286 in 
the example shown in Figure 21). 

Figure22 shows theprocessingoperationsperformed by cameratransfonnation 
calculator 70 at step S292. 

-Referring to Figure 22. at step S330, camera. transformation calculator 70 
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considers the next pair of keyframe sets which meet in the sequence 250 (this 
being the first pair the first time step S330 is performed). 

At step S3 3 2, camera transformation calculator 70 uses a fourth procedure 
5 defined by instructions stored in procedure instruction store 80 to register the 
camera transformations previously calculated for the keyframes which are 
adjacent to the boundary where the sets meet (that is, keyframe 240 and 
keyframe 242 in Figure 21). As noted previously, the camera transformations 
for the keyframes in each respective set are defined relative to different 
10 keyfirames (that is, firame 1 of the original triple of keyframes for the set). 
Accordingly, the processing performed at step S3 3 2 calculates the relationship 
between the transformations for each set so that the sets become merged. 

Figure 23 shows the processing operations performed by camera transformation 
15 calculator 70 at step S332. 

Referring to Figure 23, at step S350, camera transformation calculator 70 
considers all of the keyfirames in the first set and all of the keyfirames in the 
second set, and at step S3 52 identifies each tracked feature which has a position 
20 previously determined by feature detector and tracker 50 in at least two of the 
keyfirames firom each set. 



25 



At step S354, for each tracked feature identified at step S352, camera 
transformation calculator 70 reads the 3D position of the corresponding feature 
point which has previously been calculated using the keyfirames in the first set 
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at step S220, S274, S306 or S3 14. These 3D points are defined in a first 
coordinate system determined by the keyfi-ames in the first set. 

At step S356, camera transformation calculator 70 reads the 3D position of each 
feature point previously calculated at step S220, S274, S306 or S3 14 using the 
keyfi-ames in the second set which corresponds to a tracked feature identified 
at step S350. These 3D points are defined in a second coordinate system 
determined by the keyframes in the second set. 

The 3D points read at step S354 and the 3D points read at step S356 represent 
the same physical points on object 140. Therefore, a point in the first 
coordinate system read at step S354 and the corresponding point in the second 
coordinate system read at step S356 (that is, the point representing the same 
physical point on object 120) must have the same position in three-dimensions. 

Accordingly, at step S358, camera transformation calculator 70 uses the 3D 
positions of each pair of corresponding feature points (that is, each pair 
comprising a point in the first coordinate system and the corresponding point 
in the second coordinate system) to calculate the transformation between the 
coordinate system of the first set of keyfirames and the coordinate system of the 
second set of keyfi-ames. In this embodiment, the processing at step S358 is 
performed in a conventional manner, for example as described in "Closed Fom 
Solution of Absolute Orientation Using Unit Quaternions" by Horn in Journal 
of Optical Soc. of America4(4), pages 629-649, April 1987. 
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Referring again to Figure 22, at step S334, camera transformation calculator 70 
determines whether there is another pair of keyframe sets in the sequence 250 
which meet. Steps S330 to S334 are repeated until each pair of keyframe sets 
which meet has been processed in the manner described above, 

5 

As a result of performing the processing at step S292 (Figure 19), processing 
apparatus 20 has merged together sets of keyframe which met in the sequence 
250. Thus, referring to the example shown in Figures 21 and 24, set 276 and 
set 286 have been merged into a single set 290, in which the positions and 
10 orientations of all of the keyframes within the set are defined relative to a single 
keyframe within the set (keyframe 226). 

Referring again to Figure 19, at step S294, camera transformation calculator 70 
performs processing to merge consecutive sets of keyframes which are 
1 5 separated in the sequence 250 by other keyframes (for example set 266 and set 
290 in Figure 24). 

Figure 25 shows the processing operations performed by camera transformation 
calculator 70 at step S294. 

20 

Referring to Figure 25, at step S3 70, camera transformation calculator 70 
considers the next gap between consecutive pairs of keyframe sets, and at step 
S372 discards the keyframes within the gap (for which no camera 
transformations have previously been calculated and stored). Thus, referring 
25 to the example shown in Figure 24, at step S3 72, keyfreunes 220 and 222 are 
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discarded. 



At step S374, feature matcher 90 perfonns processing to match features in the 
keyframes adjacent the gap (that is, keyframe 218 from set 266 and keyframe 
224 from set 290 in the example shown in Figure 24). More particularly, in this 
embodiment, feature matcher 90 identifies matching features in two ways. In 
the first way, feature matcher 90 reads the positions of any feature points which 
have previously been tracked by feature detector and tracker 50 between the 
keyframes adjacent the gap and also one other keyframe in each set having the 
gap therebetween. In the second way, feature matcher 90 causes display 
processor 110 to display the keyframes which are adjacent to die gap to the user 
on display device 22. In addition, feature matcher 90 causes display processor 
1 10 to display a visual indication in each of the displayed keyframes of the 
position of each feature tracked by feature detector and tracker 50 which also 
has a recorded (tracked) position in at least one other keyframe from the set 
containing the displayed keyframe. Thus, referring to the example shown in 
Figure 24, a visual indication is displayed to die user of the position of each 
fracked feature in keyframe 218 which also has a recorded position in at least 
oneoftheother keyframes inset266. Similarly, a visual indication is displayed 
for keyframe 244 showing the position of each tracked feature which also has 
a recorded position in at least one keyframe in set 290. Feature matcher 90 
fiirther causes display processor 1 10 to display a message requesting the user 
to identify matching feature points between the two displayed keyframes for 
which an indicator is displayed. 
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Signals defining the matching features input by the user using a user input 
device 24 are stored by central controller 30 in data store 40. If the total 
number of matches identified by reading the features tracked by feature detector 
and tracker 50 and input by the user is less than a threshold, which, in this 
embodiment, is set to three, then feature matcher 90 causes display processor 
110 to request the user to identify further matching features from those 
displayed with an indicator, or to confirm that no more matching features can 
be identified. 

At step S3 76, camera transformation calculator 70 determines whether the 
number of pairs of features matched at step S374 is less than a threshold, which, 
in this embodiment, is set to three. 

If it is determined at step S3 76 that the number of matched features is greater 
than or equal to the threshold, then, at step S3 78, for each feature matched in 
the keyframes adjacent the gap at step S3 74, camera transformation calculator 
70 determines whether a feature point in 3D has already been calculated at step 
S220, S274, S306 or S3 14. If a feature point in 3D has not akeady been 
calculated for a feature matched at step S3 74, then, camera transformation 
calculator 70 uses the keyframes in the set containing a first one of the 
keyframes adjacent the gap to calculate a feature point in 3D. This processing 
is performed in the same way as the processing at step S220 (Figure 13), and 
accordingly will not be described again here. 

At step S3 80, camera transformation calculator 70 repeats the processing 
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perfonned at step S378, but this time to define a point in 3D for each feature 
matched at step S374 for which a feature point has not ateady been calculated 
using the keyframes in the set which contain the second keyframe adjacent the 
gap. 

At step S382, camera transformation calculator 70 uses the matched features to 
register the camera transformations for the keyframes adjacent the gap, and 
hence merge the sets of keyframes having the gap therebetween. The 
processing performed at step S382 is the same as that performed at step S332 
(Figure 22), which is described above with respect to Figure 23, and 
accordingly it will not be described again here. However, the processing 
performed at step S382 also takes into account the 3D points calculated at step 
S378 and step S380. 

Thus, the processing performed at step S378 and S380 requires the calculation 
of a 3D point using the position of a feature point in at least two keyframes of 
a set, and the processing perfonned at step S382 rehes on the registration of 
corresponding 3D points. It is for this reason that the feature points matched 
at step S374 comprise only points for which a position was recorded by feature 
detector and tracker 50 in at least two keyframes of each set. 

On the other hand, it is determined at step S376 that the number of matched 
features is less than the threshold, then steps S378. S380 and S382 are omitted 
so that the pair of sets of keyframes currently being considered are not merged. 
A situation in which there is not a sufficient number of matched features is 
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unlikely to arise due to the way in which processing was carried out at step S6 
(Figure 3) to define the initial triples of keyframes. However, such a situation 
could arise if, for example, camera 140 was switched off while it was moved 
to a different position during the recording of the sequence of frames 130. 

5 

At step S3 84, camera transformation calculator 70 determines whether there is 
another gap between consecutive pairs of sets of keyframes in the sequence 
250. Steps S370 to S3 84 are repeated until each consecutive pair of keyframe 
sets with a gap therebetween has been processed in the manner described above. 

10 

As a result of performing the processing at step S294 (Figure 19), processing 
apparatus 20 has merged into a single set all of the keyframes remaining in 
sequence 250, so that the relationship between the camera transformations for 
each keyframe relative to all of the other keyfiumes is known. 

15 

Thus, referring to the example shown in Figure 26, the relative positions and 
orientations of all of the keyframes remaining in sequence 250 have been 
calculated (the break 292 in the sequence occurring as a result of keyframes 220 
and 222 being discarded at step S372 (Figure 25)). 

20 . . ^ 

Consequently, referring to the example shown in Figure 27, the relative 
positions and orientations for a subset of the frames in the original sequence of 
images 130 (that is, the keyframes in the sequence 250) have now been 
calculated. 
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The method of calculating the relative positions and orientations of the input 
image frames in the manner described above in this embodiment provides a 
number of advantages. For example, by considering separate sets of keyframes 
within the sequence 250, rather than the sequence 250 as a whole, more 
accurate camera transformations can be calculated, and the accumulation of 
errors is reduced. In addition, by selecting the keyframes to form the ends of 
sets (that is, frame 1 and frame 3 of each triple) in dependence upon the input 
data, rather than using predetermined positions, and by filling in the sets (that 
is, calculating the camera transformations for intra-triple keyframes) andjoining 
sets as described above, the accuracy of the camera transformations is increased 
ftuther. 

Referring again to Figure 3, at step S14, central controller 30 outputs the 
camera transformations for the keyframes in the set 250 stored in camera 
transfonnation store 100, for example as data recorded on a storage device such 
as disk 102 or as a signal 104 (Figure 2). The positions of 3D points calculated 
by camera transformation calculator 70 (for example at steps S220, S274, S306, 
S3 14, S378 or S3 80) may also be output. 

A number of modifications are possible to the embodiment described above. 

For example, in the embodiment above, the input images comprise a temporal 
sequence of frames recorded by a video camera 140. However, the processing 
described above, can equaUy be appUed to "still" photographs recorded at 
different positions and arranged in,a.spatial sequence (that.is, an. order defined 
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by the relative positions around object 120 at which the photographs were 
taken). 

In the embodiment above, at step S2 (Figure 3) data input by a user defining the 
intrinsic parameters of camera 140 is stored. However, instead, default values 
may be assvuned for some, or all, of the intrinsic camera parameters, or 
processing may be performed to calculate the intrinsic parameter values in a 
conventional manner, for example as described in "Euclidean Reconstruction 
firom UncaUbrated Views" by Hartley in Applications of Invariance in 
Computer Vision, Mundy, Zisserman and Forsyth eds, pages 237-256, Azores 
1993, Springer- Verlag, ISBN 0387582401. 

In the embodiment above, when calcxilating camera transformations, the 
intrinsic camera parameters are combined widi the calculated camera projection 
for a keyfirame to give the position and orientation of the keyframe. However, 
the processing described above can also be applied to calculate the camera 
projections alone for the keyframes. 

When tracking features between images at step S24 and step S34 (Figure 5), a 
search for new matching pixels may be carried out in a sub-sampled image, or 
the search may be carried out in a hierarchical coarse-to-fine manner, rather 
than searching every pixel in an image. 

In the embodiment above, the features being tracked are "topped up" (that is, 
further features are identified for tracking to replace features for which track has 
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been lost) only at keyframes. However, it is possible to identify further features 
for tracking when the number of tracked features falls below a threshold. 

In the embodiment above, at steps S28 and S40 (Figure 5), a calculated 
uncertainty score for an image frame is compared against a threshold value and, 
if the uncertainty score is less than the threshold value, the image frame is set 
as a keyframe. However, instead of using a threshold value, processing may be 
perfomed to set an image frame as a keyframe if the uncertainty score for the 
frame is a local minimum with respect to the uncertainty scores of the 
surrounding image frames. 

In the embodiment above, at step S4 (Figure 3), keyframes are selected from the 
input frames. This reduces the amount of processing to be earned out by 

selectingasubsetofthe original frames forfiirtherprocessing(the frames in the 
subset being selected as the frames which are likely to enable accurate camera 
transforations to be calculated). However, instead, all of the input frames 
may be processed using the techniques described above. Alternatively, the 
input frames may be sub-sampled to select every "n"th frame before performing 
the processing in the embodiment above at step S4 to select keyframes. 

In the embodiment above, at step S6. triples of frames are selected, and at step 
S8, relative camera transformations are calculated for each triple. However, 
instead of using triples of frames, o Aer nmnbers of frames may be used in these 
steps.: For example "doubles" of frames may be used (for example by omitting 
steps S I48 to 8152 in Figure 9 at which the middle frame of a triple is set). 
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After calculating relative transformations for a triple of keyframes at step S8, 
the accuracy of the ceilculated transformations may be increased by performing 
the processing described above with respect to steps S270, S272 and S274 
(Figure 16) using a window which covers the three keyframes in the triple. 

In the embodiment above, during the processing performed at step S294 (Figure 
19) to merge sets of keyframes with a gap therebetween, features are matched 
in the keyframes adjacent the gap (step S3 74 in Figure 25), As part of the 
processing to match features, the keyframes are displayed to a user and the user 
is requested to identify matching feature points. However, rather than 
requesting the user to match features, automatic feature matching may be 
performed by processing apparatus 20 to match features in the keyframes 
adjacent the gap in a conventional maimer, for example as described in 
"Adaptive Least Squares Correlation: A Powerfiil Image Matching Technique" 
by A.W. Gruen in Photogrammetry Remote Sensing and Cartography, 14(3), 
1985, pages 175-187, 

In the embodiment above, when adjusting the camera transformations for a 
given set of keyframes at step S206 (Figure 12) the keyframes within a window 
270 (Figures 15a-15d) are considered. The window has a constant size and is 
moved at each stage of the processing by one keyframe to consider a difiFerent 
subset of the keyframes within the set. However, the window 270 may be 
moved by more than one keyframe at a tune provided that the number of 
keyframes moved is not greater than the size of the window itself. More 
particularly, the window 270 may still be moved sequentially through the 
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keyframes within the set, but instead of moving the window one keyframe at a 
time (that is, to take in one new keyframe and to drop out one keyframe already 
processed), the window could be moved to take in any number of new 
keyframes up to the number of keyframes that can fit within the window. In 
this way, each keyframe still appears in the window at least one (each keyframe 
appearing only once if the window is moved so that there is no overlap between 
adjacent positions). 

In addition, the size of the window may be changed as it is moved. For 
example, as the number of tracked feature points changes as the window 270 
is moved through the keyframes in the set, the size of the window 270 may be 
adjusted to keep the processing to be earned out substantially constant (that is, 
to increase the size of the window as the number of tracked features decreases, 
or to decrease the size of the window as the number of tracked features 
increases). 

Further, after the window 270 has been moved sequentiaUy to consider each 
subset of the keyframes within the set, the window 270 may then be moved 
sequentially back through the keyframes within the set, and this forwards and 
backwards movement may be repeated until the solution for the calculated 
camera transformation and 3D points converges. That is, the window 270 may 
be moved as shown in Figure 15a, 15b, 15c and 15d, and then moved 
backwards as shown in Figure 15d, 15c, 15b and 15a, and diis movement may 
be repeated: 
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In the embodiment above, at step S272 (Figure 16) the error measm-ement value 
to be minimised (equation 21) is calculated on the basis, of the measured 
position of every tracked feature point within the keyframes within the window 
270 and every keyframe within the window 270. However, instead, the error 
measurement value may be calculated using different combinations of tracked 
feature points and keyframes. For example, the error measurement value may 
be calculated using the measured position in keyframes within the window of 
each tracked feature point which does not have a measured position in 
keyframes within the set which are outside the window 270 (such as feature 
point 2 and feature point 3 in the example shown in Figure 17), together with 
each keyframe which is within window 270. In this way, the tracked feature 
points which contribute to the error measurement value are the same as the 
tracked feature points for which the 3D positions ("X" in equation 21) are 
varied at step S274 to minimise the error measurement value. 

In the embodiment above, the error measurement value set at step S272 (Figure 
16) is given by equation (21). However, different representations of the error 
between estimated feature locations (based on calculated camera 
transformations) and measured feature locations can be used. For example, 
in equation (21) could be allowed to take different values for different values 
of "i" and "j". This allows the error associated with an image measurement to 
vary, but only such that the x and y components are the same (i.e. the error is 
circular). In cases where an ellipsoidial error is associated with an image 
measurement, the error measurement value, E(P,X) can be calculated by: 



78 



2- 

263590K 



(x.) 



..(27) 



where: (Wjj)^ is the inverse covariance matrix, which is a 2 by 2 matrix, 
with a and P labelling the rows and columns; and 



the other symbols are as defined previously for equation (2 1). 

Further, rather than using the sum of squares of differences between projected 
and measured image positions as in equations (21) and (27) above, a different 

fiinctionofthe differences betweenprojectedandmeasuredimagepositions can 
be used. For example. E(P,X) can be calculated as: 



^(^^=Ep 



(r.) 



..(28) 



where p defines the type of fimction which the image measurement errors take 
(and is defined to include the weightings, W) and the other symbols are 
as defined previously for equation (21). For example, the foUowing 
fimction p(x) may be used, as suggested in P.J. Huber in "Robust 
Statistics" Wiley, New York, 1981, ISBN 0471418056: 



a\x\ -0.5o^ otherwise 



..(29) 
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In the embodiment above, the processing perfomied at step S206 (Figure 12) to 
adjust the camera transformations for a given window 270 of keyframes is 
carried out on the basis of a calibrated perspective camera transformation 
model. That is, "P" in equation (21) is given by P K[R,t], where "K" defines 
the known intrinsic camera parameters, as described above. However, the 
processing can be applied to other camera models. For example: 

(i) In the case of a perspective camera transformation model in which 
the entries of "K" are known only approximately, then, as well as 
varying the values of "P" and "X" when E(P,X) is minimised at step 
S274, the values of the entries in "K" representing the focal length, 
principal point and/or first order radial distortion coefficient may be 
varied as well (the aspect ratio and skew angle being constant if the 
images are recorded with the same camera 140). 

(ii) In the case of a projective camera model, in which the intrinsic 
camera parameters are not known, then "P" is a 3 by 4 projection 
matrix defined up to an overall scale. In this case, the entries of "P" 
are all varied independently when step S274 is performed, (The 
intrinsic camera parameters may then be calculated using a 
conventional self-calibration technique, for example as described in 
"Euclidean Reconstruction from Uncalibrated Views" by Hartley in 
Applications of Invariance in Computer Vision, Mundy, Zisserman 
and Forsyth eds, pages 237-256, Azores 1993.) 
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In the case of a weak perspective camera transformation model, "P 
has the restricted form: 



P = K rI t 
2 y 




....(30) 



0 z' 



where R, and are the first and second colmnn vectors 
respectively which make up the 3 by 3 rotation matrix R; 

1;, and ty are the translations in the "x" and "y" directions 
respectively; 

Z"^ is the average distance of the points in the real-world 
scene which are recorded in the image from the focal 
plane of the camera. 

In this case, the term \^^), is constant. Accordingly, iterative 
adjustment at step S274 using a non-linear least squares method is 
mmecessaiy, and instead, a closed form solution can be calculated 
based on factorisation (by finding the best decomposition of the (xjj) 
matrix into die product of a P; matrix and Xj matrix), in a 
conventional manner, for example as described in "Shape and 
Motion from Image Streams under Orthography: a Factorization 
Method" by Tomasi and Kande, UCV 9:2, 137-154 (1992). 
However, the calciilation is still applied to the window 270 of 



81 



263590X 



keyframes, which is then moved sequentially to consider all of the 
keyframes within the set, in the manner described above, 

(iv) In the case of an affine camera transformation model, "P" is only 

defined up to a scale, and corresponds to a projective camera with its 
optical centre at the plane at infinity. More particularly, "P" is a 3 by 
4 matrix, but some of the entries are 0 as follows: 





(p 

^ 11 


^.2 


^.3 




p = 




P 


^23 


^24 




0 


0 


0 


^34; 



In this case, the term (PiXj)3 is constant Accordingly, iterative 
adjustment at step S274 using a non-linear least sqxiares method is 
uimecessary, and instead, a closed form solution can be calculated 
based on factorisation (by finding the best decomposition of the (Xij) 
matrix into the product of a P; matrix and Xj matrix), in a 
conventional maimer, for example as described in "Shape and 
Motion from Image Streams under Orthography: a Factorization 
Method" by Tomasi and Kande, IJCV 9:2, 137-154 (1992). 
However, the calculation is still applied to the window 270 of 
keyframes, which is then moved sequentially to consider all of the 
keyframes within the set, in the manner described above. 

In the embodiment above, at step S274, iterations of the minimisation algorithm 
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are stopped when the value of A exceeds a threshold. However, the iterations 
may be stopped when the number of iterations exceeds a threshold, or in 
dependence upon a combination of the value of A and the number of iterations. 

In the embodiment above, at step S274, the error measurement value is 
minimised using a Levenberg-Marquardt algorithm. However, different 
algorithms may be used to minimise the error measurement value. For example, 
a non-linear least squares method such as the Gauss-Newton method may be 
used. 

In the embodiment above, at step S 1 0 (Figure 3), the camera ixansfonnations are 
calculated for intra-triple keyframes and, subsequently, at step S12, the 
keyframe sets created at step SIO are expanded and joined together. However, 
step S 12 may be carried out before step SIO. 

In the embodiment above, when creating and expanding sets of keyframes at 
steps S8. SIO and S12 (Figure 3) aU of the sets are processed in one step before 
the next step is performed. More particularly, at step S8, camera 
transformations are calculated for each triple and at step SIO, camera 
transformations are calculated for the intra-triple keyframes in each triple, 
before each set is extended at step S290 (Figure 19) during the processing 
performed at step S12. However, instead, processing may be performed on a 
set-by-set basis, for example as follows: 

(a) . calculateandtestcameratransformationsforthenexttripleofkeyframes 
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using the processing at step S8 for one triple; 

(b) calculate camera transformations for intra-triple keyframes within the 
triple processed at step (a) using the processing at step SIO for one 
triple; 

(c) calculate camera transformations for keyframes between the triple 
processed at step (a) and the preceding and succeeding triples in the 
sequence using the processing at step S290 to extend the set until the 
accuracy of the calculated transformations falls below the threshold (step 
S3 10 in Figure 20); 

(d) repeat steps (a) to (c) for the next triple of keyframes, and repeat until aU 
triples have been processed; 

(e) merge the sets of keyframes as described above with respect to steps 
S292 and S294 in Figure 19. 

In the embodiment above, at step S206 (Figure 12) and step S3 14 (Figure 20) 
calculated camera transformations are adjusted. However, it is not essential that 
these steps are performed, although if the steps are performed, then the overall 
accuracy of the calculated camera transformations is increased. 

In the embodiment above, processing is performed by a computer ixsing 
processing routines defined by programming instructions. However, some or 
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all, of the processing could be performed using hardware. 

The present application incorporates by cross-reference the full contents of the 
applicant's following co-pending applications: 9927875.6 (attorney ref 
5 2636501); 9927906.9 (attorney ref 2641601); 9927907.7 (attorney ref 
2641701); 9927909.3 (attorney ref: 2641901); 9929957.0 (attorney ref: 
2641001); 0001300.3 (attorney ref: 2636001); 0001479.5 (attorney ref: 
2636201) and 0018492.9 (attorney ref: 2635801). 

0 In the embodiments described above, step S374 (Figure 25) may be carried out 
usmg the techniques described in application 9927906.9, 9927907.7 and/or 
9927909.3. 



85 



2635907^ 



ANfNEX A - 

1. CORNER DETECTION 

1.1 Summary 

This process described below calculates comer points, to sub-pixel accuracy, 
from a single grey scale or colour image. It does this by first detecting edge 
boimdaries in the image and then choosing comer points to be points where a 
strong edge changes direction rapidly. The method is based on the facet model 
of comer detection, described in Haralick and Shapiro*. 

1.2 Algorithm 

The algorithm has four stages: 

(1) Create grey scale image (if necessary); 

(2) Calculate edge strengths and directions; 

(3) Calculate edge boimdaries; 

(4) Calculate comer points. 

1.2.1 Create grey scale image 



The comer detection method works on grey scale images. For colour images, 
the colour values are first converted to floating point grey scale values using the 
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formula: 

greyjcale = (0.3 x red) + (0.59 x green) + (O.H x blue) ....A-I 

This is the standard definition of brightness as defined by NTSC and described 
in Foley and van Dam". 

1-2.2 Calculate edge strengths and directions 

The edge strengths and directions are calculated using the 7x7 integrated 
directional derivative gradient operator discussed in section 8.9 of HaraUck and 
Shapiro'. 

The row and column forms of the derivative operator are both applied to each 
pixel in the grey scale image. The results are combined in die standard way to 
calculate the edge strength and edge direction at each pixel. 

The output of this part of the algorithm is a complete derivative image. 
1-2.3 Calculate edge boundarie.s 

The edge boundaries are calculated by using a zero crossing edge detection 
method based on a set of 5x5 kernels describing a bivariate cubic fit to the 
neighbourhood of each pixel. 



edge boundary detection method peaces an edge at all pixels which 
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close to a negatively sloped zero crossing of the second directional derivative 
taken in the direction of the gradient, where the derivatives are defined using 
the bivariate cubic fit to the grey level surface. The subpixel location of the zero 
crossing is also stored along with the pixel location. 

The method of edge boundary detection is described in more detail in section 
8,8.4 of Haralick and Shapiro*. 

1.2.4 Calculate comer points 

The comer points are calcidated using a method which uses the edge boundaries 

calculated in the previous 

step. 

Comers are associated with two conditions: 

(1) the occurrence of an edge boundary; and 

(2) significant changes in edge direction. 

Each of the pixels on the edge boimdary is tested for "comemess" by 
considering two points equidistant to it along the tangent direction. If the 
change in the edge direction is greater than a given threshold then the point is 
labelled as a comer. This step is described in section 8.10.1 of Haralick and 
Shapiro'. 
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Finally the comers are sorted on the product of the edge strength magnitude and 
the change of edge direction. The top 200 comers which are separated by at 
least 5 pixels are output. 

2- 3D SURFArF GENERATTONJ 
21 Architecture 

In the method described below, it is assumed that the object can be segmented 
from the background in a set of images completely surrounding the object. 
Although this restricts the generality of the method, this constraint can often be 
arranged in practice, particularly for small objects. 

The method consists of five processes, which are run consecutively: : 

First, for aU the images in which the camera positions and orientations 
have been calculated, the object is segmented from the background, 
using colour information. This produces a set of binary images, where 
the pixels are marked as being either object or background. 

The segmentations are used, together with the camera positions and 
orientations, to generate a voxel carving, consisting of a 3D grid of 
voxels enclosing the object. Each of the voxels is marked as being either 
object or empty space. 
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The voxel carving is turned into a 3D surface triangulation, using a 
standard triangulation algorithm (marching cubes). 

The number of triangles is reduced substantially by passing the 
5 triangulation through a decimation process. 

Finally the triangulation is textured, using appropriate parts of the 
original images to provide the texturing on the triangles. 

10 2.2 Segmentation 

The aim of this process is to segment an object (in jfront of a reasonably 
homogeneous coloured background) in an image using colour information. The 
resulting binary image is used in voxel carving. 

15 

Two alternative methods are used: 

Method 1 : input a single RGB colour value representing the backgroxmd 
colour - each RGB pixel in the image is examined and if the Euclidean 
20 distance to the background colour (in RGB space) is less than a specified 

threshold the pixel is labelled as background (BLACK). 

Method 2: input a "blue" image containing a representative region of the 
background, 

25 
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The algorithm has two stages: 

( 1) Build a hash table of quantised background colours 

(2) Use the table to segment each image. 
Step 1) Build hash table 



Go through each RGB pixel, p, in the "blue" background 



miage. 



Set q to be a quantised version of p. Explicitly: 
q = (p+tl2)lt 

where t is a threshold determining how near RGB values need to be to 
background colours to be labelled as background. 

The quantisation step has two effects: 

1) reducing the number of RGB pixel values, thus increasing the efficiency 
of hashing; 

2) defining the threshold for how close a RGB pixel has to be to a 
background colour pixel to be labelled as background. 

q is now added to a hash table (if not akeady in the table) using the (integer) 
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hashing function 

„..A-3 

That is, the 3 least significant bits of each colour field are used. This function 
is chosen to try and spread out the data into the available bins. Ideally each bin 
in the hash table has a small nimiber of colour entries. Each quantised colour 
RGB triple is only added once to the table (the frequency of a value is 
irrelevant). 

Step 2) Segment each image 

Go through each RGB pixel, v, in each image. 

Set w to be the quantised version of v as before. 

To decide whether w is in the hash table, explicitly look at all the entries in the 
bin with index h(w) and see if any of them are the same as w. If yes, then v is 
a background pixel - set the corresponding pixel in the output image to BLACK. 
If no then v is a foreground pixel - set the corresponding pixel in the output 
image to WHITE 

Post Processing: For both methods a post process is performed to fill small 
holes and remove small isolated regions. 
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A median filter is used with a circular window. (A circular window is chosen 
to avoid biasing the result in the x or y directions). 

Build a circular mask of radius r. Explicitly store the start and end values for 
each scan line on the circle. 

Go through each pixel in the binary image. 

Place the centre of the mask on the cuixent pixel. Count the number of BLACK 
pixels and the number of WHITE pixels in the circular region. 

If (#WHITE pixels > #BLACK pixels) then set corresponding output pixel to 
WHITE. Otherwise output pixel is BLACK. 

2.3 Voxel carving 

The aim of this process is to produce a 3D voxel grid, enclosing the object, with 
each of the voxels marked as either object or empty space. 

The input to the algorithm is: 

a set of binary segmentation images, each of which is associated with a 
camera position and orientation; 

- 2 sets of 3D co-ordinates, (xmin, ymin, zmin) and (xmax, ymax, zmax). 
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describing the opposite vertices of a cube surrounding the object; 

a parameter, n, giving the number of voxels required in the voxel grid. 

A pre-processing step calculates a suitable size for the voxels (they are cubes) 
and the 3D locations of the voxels, using n, (xmin, ymin, zmin) and (xmax, 
ymax, 23nax). 

Then, for each of the voxels in the grid, the mid-point of the voxel cube is 
projected into each of the segmentation images. If the projected point falls onto 
a pixel which is marked as background, on any of the images, then the 
corresponding voxel is marked as empty space, othenvise it is marked as 
belonging to the object. 

Voxel carving is described further in "Rapid Octree Construction from Image 
Sequences" by R. Szeliski in CVGIP: Image Understanding, Volume 58, 
Number 1, July 1993, pages 23-32. 

2.4 Marching cubes 

The aim of the process is to produce a surface triangulation from a set of 
samples of an implicit function representing the surface (for instance a signed 
distance function). In the case where the implicit function has been obtained 
from a voxel carve, the implicit ftmction takes the value -1 for samples which 
are inside the object and +1 for samples which are outside the object. 
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Marching cubes is an algorithm that takes a set of samples of an implicit surface 
(e.g. a signed distance function) sampled at regular intervals on a voxel grid, 
and extracts a triangulated surface mesh. Lorensen and Cline"' and 
BloomenthaP^ give details on the algorithm and its implementation. 

The marching-cubes algorithm constructs a surface mesh by "marching" 
around the cubes while following the zero crossings of the implicit surface 
f(x)=0, adding to the triangulation as it goes. The signed distance allows the 
marching-cubes algorithm to interpolate the location of the surface with higher 
accuracy than the resolution of the volume grid. The marching cubes algorithm 
can be used as a continuation method (i.e. it finds an initial surface point and 
extends the surface fi:om this point). 

2-5 Decimation 

The aim of the process is to reduce the number of triangles in the model, 
making the model more compact and therefore easier to load and render in real 
time. 

The process reads in a triangular mesh and then randomly removes each vertex 
to see if the vertex contributes to the shape of the surface or not. (i.e. if the hole 
is filled, is the vertex a "long" way firom the filled hole). Vertices which do not 
contribute to the shape are kept out of the triangulation. This results in fewer 
vertices (and hence triangles) in the fmal model. 
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The algorithm is described below in pseudo-code. 
INPUT 

Read in vertices 

Read in triples of vertex IDs making up triangles 

PROCESSING 

Repeat NVERTEX times 

Choose a random vertex, V, which hasn't been chosen before 
Locate set of all triangles having Vas a vertex, S 
Order S so adjacent triangles are next to each other 
Re-triangulate triangle set, ignoring V (i.e. remove selected triangles 
& Vand then fill in hole) 

Find the maximum distance between Vand the plane of each triangle 
If (distance < threshold) 

Discard Vand keep new triangulation 

Else 

Keep Vand return to old triangulation 

OUTPUT 

Output list of kept vertices 
Output updated list of triangles 

The process therefore combines adjacent triangles in the model produced by the 
marching cubes algorithm, if this can be done without introducing large errors 
into the model. 
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The selection of the vertices is carried out in a random order in order to avoid 
the effect of gradually eroding a large part of the surface by consecutively 
removing neighbouring vertices. 

2-6 Furdier Sur face Generation Techniq ue-; 

Further techniques which may be employed to generate a 3D computer model 
of an object surface include voxel colouring, for example as described in 
"Photorealistic Scene Reconstruction by Voxel Coloring" by Seitz and Dyer in 
Proc. Conf. Computer Vision and Pattern Recognition 1997, pl067-1073, 
"Plenoptic Image Editing" by Seitz and Kutulakos in Proc. 6th International 
Conference on Computer Vision, pp 17-24, "What Do N Photographs TeU Us 

AboutSDShape?" by Kutulakos andSeitz in University ofRochesterComputer 
Sciences Technical Report 680, January 1998, and "A Theory of Shape by 
Space Carving" by Kutulakos and Seitz in University of Rochester Computer 
Sciences Technical Report 692, May 1998. 

3. TEXTURING 

The aim of the process is to texture each surface polygon (typically a triangle) 
with the most appropriate image texture. The output of the process is a VRML 
model of the surface, complete with texture co-ordinates. 

The triangle having the largest projected area is a good triangle to use for 
texturing, as it is the triangle for- which, the texture will appear at highest 
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resolution. 

A good approximation to the triangle with the largest projected area, under the 
assumption that there is no substantial difference in scale between the different 
images, can be obtained in the following way. 

For each surface triangle, the image "i" is foimd such that the triangle is the 
most front facing (i.e. having the greatest value for n^.Vi, where n^ is the triangle 
normal and Vj is the viewing direction for the "i" th camera). The vertices of the 
projected triangle are then used as texture co-ordinates in the resxilting VRML 
model. 

This technique can fail where there is a substantial amount of self-occlusion, or 
several objects occluding each other. This is because the technique does not 
take into accoxmt the fact that the object may occlude the selected triangle. 
However, in practice this does not appear to be much of a problem. 

It has been found that, if every image is used for texturing then this can result 
in very large VRML models being produced. These can be cumbersome to load 
and render in real time. Therefore, in practice, a subset of images is used to 
texture the model. This subset may be specified in a configuration file. 
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CLAIMS 



1. A method of processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
5 camera projection for each of at least some of the images defining the 
projection of points on the object surface into the image, the method 
comprising: 

a selecting step comprising using the matching features to select a first 
image in the sequence and a second image in the sequence for each of a 
10 plurality of sets of images to give initial end images in the sequence for each 
set; 

a set forming step comprising, for the selected pair of end images for 
each set, using the matching features to calculate a camera projection for one 
of the images in the pair relative to the other image in the pair, to form a set 
15 thereform; 

a set expanding step comprising increasing the nimiber of images in at 
least some of the sets by using the matching features to calculate a respective 
camera projection for each of at least some of the images lying in the sequence 
between end images of a set and each of at least some of the images lying in 
20 the sequence between consecutive sets, each camera projection being 
calculated relative to the images in a set; and 

a set merging step comprising using the matching features to determine 
the relationship between the camera projections calculated for the different 
sets, thereby to merge the sets. 
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2. A method according to claim 1, wherein, in the selecting step, an image 
to form an end image is selected in dependence upon the number of matching 
features in the image which are also present in the preceding end image in the 
sequence. 

3. A method according to claim 1 or claim 2, wherein, in the set 
expanding step, die camera projections for images lying between the end 
images for a given set are calculated before the camera projections for images 
lying between the given set and adjacent sets in die sequence are calculated. 

4. A method according to any preceding claim, wherein, in the step of 
calculating camera projections for images lying between consecutive sets: 

(i) the selected end images are prioritised in dependence upon an 
accuracy of the calculated camera projections of the end images; 

(ii) the end image with the highest priority is selected; 

(iii) the camera projection for the image in the sequence adjacent the 
selected end image is calculated relative to the selected end image to form a 
new end image; 

(iv) the priority of the end images is updated in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) steps (ii) to (iv) are repeated. 

5. A method according to any of claims 1 to 3, wherein pairs of end 
images selected in the selecting step are considered in turn and, for a given 
pair of end images, the set forming step to form a set, therefrom and the set^ 
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expanding step to calculate a respective camera projection for at least one 
image lying in the sequence between the pair of end images and at least one 
image lying in the sequence between an end image of the pair and an end 
image for a consecutive set, are performed before performing the set forming 
step and the set expanding step for the next pair of end images to be 
considered. 

6. A method according to any preceding claim, further comprising the step 
of generating a signal conveying the calculated camera projections. 

7. A method according to claim 6, further comprising the step of recording 
the signal either directly or indirecdy. 

8. A method according to any preceding claim, wherein the sequence of 
images is a temporal sequence in which the images are arranged in time-of- 
recording order. 

9. A method according to any preceding claim wherein the images 
comprise video images. 

10. A method according to any preceding claim, further comprising the step 
of processing image data defining the images in the sequence to generate the 
data defining the matching features. 

n . A method according to any preceding claim, further comprising the step 
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of using the matching features and the calculated camera projections to 
generate data defining a 3D computer model of the object in the images. 

12. A method according to claim 11, further comprising the step of 
generating a signal conveying the 3D computer model. 

13. A method according to claim 12, further comprising the step of 
recording the signal either directly or indirectly. 

14. A method of processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining a projection 
of the object into liie image, the method comprising: 

selecting a pluraUty of groups of the images, each group comprising at 
least two images and at least one pair of consecutive groups in the sequence 
having images in the sequence therebetween which are not part of a group; 

forming a set of images fi:om each selected group by calculating a 

respective cameraprojectionforeachimage in the group relative toareference 
image in the group; 

calculating a respective camera projection for each of at least some of 
the images in the sequence not selected to form a group relative to a reference 
image in a group; and 

calculating the relationship between the camera projections calculated 
for each group. 
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15. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining the 
projection of points on the object surface into the image, comprising: 

selecting means for using the matching features to select a first image 
in the sequence and a second image in the sequence for each of a plurality of 
sets of images to give initial end images in the sequence for each set; 

set forming means for using the matching features to calculate, for the 
selected pair of end images for each set, a camera projection for one of the 
images in the pair relative to the other image in the pair, to form a set 
thereform; 

set expanding means for increasing the mmiber of images in at least 
some of the sets by using the matching features to calculate a respective 
camera projection for each of at least some of the images lying in the sequence 
between end images of a set and each of at least some of the images lying in 
the sequence between consecutive sets, each camera projection being 
calculated relative to the images in a set; and 

set merging means for using the matching features to determine the 
relationship between the camera projections calculated for the dififerent sets, 
thereby to merge the sets. 

16. Apparatus according to claim 15, wherein the selecting means is 
arranged to select an image to form an end image in dependence upon the 
number of matching features in the image which are also present in the 
preceding end image in the sequence. 
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1 7. Apparatus according to claim 1 5 or claim 16, wherein the set expanding 
means is arranged to calculate the camera projections for images lying 
between the end images for a given set before calculating the camera 
projections for images lying between the given set and adjacent sets in the 
sequence. 

18. Apparatus according to any of claims 15 to 17, wherein the set 
expanding means is arranged to calculate camera projections for images lying 
between consecutive sets by: 

(i) prioritising the selected end images in dependence upon an accuracy 
of the calculated camera projections of the end images; 

(ii) selecting die end image with the highest priority; 

(iii) calculating the camera projection for the image in the sequence 
adjacent the selected end image relative to the selected end image to fonn a 
new end image; 

(iv) updating the priority of the end images in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) repeating steps (ii) to (iv). 

19. Apparatus according to any of claiins 15 to 17, wherein the apparatus 
is arranged to operate so diat pairs of end images selected by die selecting 
means are considered in turn and, for a given pair of end images, the set 
forming means is arranged to form a set dierefrom and the set expanding 
means is arranged to calculate a respective camera projection for at least one 
miage lying in the sequence between the pair of end images arid at-least 
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image lying in the sequence between an end image of the pair and an end 
image for a consecutive set, before forming a set and expanding the set for the 
next pair of end images to be considered. 

20. Apparatus according to any of claims 15 to 19, wherein the sequence 
of images is a temporal sequence in which the images are arranged in time-of- 
recording order. 

21. Apparatus according to any of claims 15 to 20, wherein the images 
comprise video images. 

22. Apparatus according to any of claims 15 to 21, further comprising 
means for processing image data defining the images in the sequence to 
generate the data defining the matching features. 

23 . Apparatus according to any preceding claim, fiirther comprising means 
for using the matching features and the calculated camera projections to 
generate data defining a 3D computer model of the object in the images. 

24. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining a projection 
of the object into the image, comprising: 

means for selecting a plurality of groups of the images, each group 
comprising at least two images and at least one pair of consecutive groups in 
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the sequence having images in the sequence therebetween which are not part 
of a group; 

means for forming a set of images from each selected group by 
calculating a respective camera projection for each image in the group relative 
to a reference image in the group; 

means for calculating a respective camera projection for each of at least 
some of the images in the sequence not selected to form a group relative to a 
reference image in a group; and 

means for calculating the relationship between the camera projections 
calculated for each group. 

25. A storage device storing instructions for causing a programmable 
processing apparatus to become operable to perform a method as set out in at 
least one of claims 1 to 14. 

26. A signal conveying instructions for causing a programmable processing 
apparatus to become operable to perform a method as set out in at least one of 
claims ! to 14. 

27. In an image processing method for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defming the projection of 
the object surface into the images, and using the matching features and the 
calculated camera projections to generate data defming the three-dimensional ■ 
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computer model of the object, an improvement comprising performing the step 
of using the matching features to determine the camera projections by : 

using the matching features to select a first image in the sequence and 
a second image in the sequence for each of a plurality of sets of images to give 
initial end images in the sequence for each set; 

for the selected pair of end images for each set, using the matching 
features to calculate a camera projection for one of the images in the pair 
relative to the other image in the pair, to form a set thereform; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at 
least some of the images lying between pairs of end images in the sequence 
and each of at least some of the images lying between consecutive sets in the 
sequence, each camera projection being calculated relative to the images in a 
set; and 

using the matching features to determine the relationship between the 
camera projections calculated for the different sets, thereby to merge the sets. 

28. In an image processing apparatus for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defining the projection of 
the object into the images, and using the matching features and the calculated 
camera projections to generate data defining the three-dimensional computer 
model of the object, a method of performing the processing to determine the 
camera projections by: 
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selecting a plurality of groups of the images, each group comprising at 
least two images and at least one pair of consecutive groups in the sequence 
having images in the sequence therebetween which are not part of a group; 

forming a set of images from each selected group by calculating a 
respective camera projection for each image in the group relative to a reference 
image in the group; 

calculating a respective camera projection for each of at least some of 
the images in the sequence not selected to form a group relative to a reference 
image in a group; and 

calculating the relationship between the camera projections calculated 
for each group. 

29. In an image processing apparatus having a processor for processing 
image data defining a sequence of images of at least one object to generate a 
three-dimensional computer model of the object by matching features in the 
images, using the matching features to determine camera projections defining 
the projection of the object surface into the images, and using the matching 
features and the calculated camera projections to generate data defming the 
three-dimensional computer model of the object, an improvement wherein the 
processor is arranged to use the matching features to determine the camera 
projections by: 

using the matching features to select a fu-st image in the sequence and 
a second image in the sequence for each of a plurality of sets of images to give 
initial end images in the sequence for each set; 

for the selected pair of end images for each set, using die matching 
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features to calculate a camera projection for one of the images in the pair 
relative to the other image in the pair, to form a set thereform; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at 
5 least some of the images lying between pairs of end images in the sequence 
and each of at least some of the images lying between consecutive sets in the 
sequence, each camera projection being calculated relative to the images in a 
set; and 

using the matching features to determine the relationship between the 
10 camera projections calculated for the different sets, thereby to merge the sets. 
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ABSTRACT 
IMAGE PROCRSSmr, APPARATUS 
In an image processing apparatus 20 an input sequence 130 of video images 
IS processed to determine the different positions and orientations at which the 
images were recorded in an efficient and accurate manner. A subset of the 
input images are selected as keyfirames to form a sequence 250 of keyframes. 
Respective triples of keyframes having different, non-overlapping positions in 
the sequence 250 are selected and processed to determine the relative positions 
and orientations at which the keyframes in each triple were recorded to form 
respective sets of keyframes. The positions and orientations of keyframes 
between the keyframes in each triple are then calculated to form expanded sets 
of keyframes 266. 276, 286. The sets are further expanded by calculating the 
positions and orientations of keyframes which lie between sets in the sequence 
250. The sets are merged by calculating the relationship between the 
coordinate systems in which the positions and orientations of the keyframes 
in each set are defined. During the processing, the positions and orientations 
calculated for keyframes in a set are adjusted to optimise the calculated 
solutions. This is performed in stages, considering at each stage a different 
window 270 of the keyframes and performing processing to minimise the error 
associated with the keyframes in the window. The window is moved 
sequentially through the keyframes so that every keyframe in a set is 
considered at least once. 
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STORE INPUT DATA 
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EACH RESPECTIVE TRIPLE 



FOR EACH TRIPLE OF 
KEYFRAMES WITH 
ADDITIONAL KEYFRAMES 
BETWEEN THE KEYFRAMES 
OF THE TRIPLE. CALCULATE 
RELATIVE CAMERA 
TRANSFORMATIONS FOR 
THE ADDITIONAL 
KEYFRAMES. AND 
REFINE THE CAMERA 
TRANSFORMATIONS FOR 
THE SET 
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JOIN THE EXISTING SETS OF 
KEYFRAMES 
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OUTPUT DATA 



S2 



S6 
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S14 



Fig 
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READ THE NUMBER OF INPUT FRAMES 

AND SET THRESHOLD VALUE FOR 
REDUCING THE NUMBER OF FRAMES 
TO BE PROCESSED 



S20 



IDENTIFY FEATURES IN THE FIRST 
INPUT FRAME TO BE TRACKED 



S22 



PERFORM TRACKING OF THE 
FEATURES FOR THE NEXT FRAME 



S24 



CALCULATE AN UNCERTAINTY SCORE 
FOR THE TRACKED FEATURES IN THE 
FRAME 



S26 




IDENTIFY FURTHER FEATURES IN THE 
NEW KEYFRAME TO BE TRACKED 
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Fig 
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(A) 



jr 

PERFORM TRACKING OF THE 
FEATURES FOR THE NEXT FRAME 



S34 



S36 

IS THE 
FRAME SEPARATED 
FROM THE PRECEDING 
^KEYFRAME BY MORE THAN THE. 
THRESHOLD NUMBER 
OF FRAMES? 



NO 
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YES 
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CALCULATE AN UNCERTAINTY SCORE 
FOR THE TRACKED FEATURES IN THE 
FRAME 
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S38 



IS THE 

UNCERTAINTY SCORE LESS"" 
THAN A THRESHOLD 
VALUE? 
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NO 





YES 
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SET FRAME AS A KEYFRAME 
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REFINE THE POSITIONS OF THE 
TRACKED FEATURES IN THE NEW 
KEYFRAME 
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READ CURRENT PARAMETERS OF THE 
FEATURE 




PREDICT THE VALUE OF THE 4D STATE 
AND STATE COVARIANCE MATRIX OF 
THE FEATURE FOR THE NEW FRAME 
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YES 
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DISCARD 




FEATURE SO 




THAT IT IS NO 
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LONGER 




TRACKED 





IS THE 
PREDICTED 
POSITIONAL UNCERTAINTY 
GREATER THAN A 
/THRESHOLD?^ 



NO 
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SEARCH A WINDOW IN THE NEW 
FRAME TO IDENTIFY PIXELS WHICH 
MATCH THE FEATURE 
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NO 
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2^ 



UPDATE THE 4D STATE 
OF THE FEATURE. THE 

STATE COVARIANCE 
MATRIX AND THE 

KALMAN FILTER GAIN 




S82 



UPDATE THE 4D STATE 
OF THE FEATURE. THE 
STATE COVARIANCE 

MATRIX AND THE 
KALMAN FILTER GAIN 
USING THE IDENTIFIED 
MATCH 



Fig. 6 
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READ THE POSITIONS OF THE NEXT 
TRACKED FEATURE IN THE NEW 
KEYFRAME AND THE PRECEDING 
KEYFRAME 
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DEFINE A WINDOW AROUND THE 
FEATURE IN EACH KEYFRAME 



PERFORM ADAPTIVE LEAST SQUARES 
CORRELATION OF THE PIXELS WITHIN 
THE WINDOW OF THE NEW KEYFRAME 
WITH THE PIXELS WITHIN THE 
WINDOW OF THE PRECEDING 
KEYFRAME 
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S104 



DISCARD THE 
MEASUREMENTS FOR 
THE FEATURE IN THE 
NEW KEYFRAME AND 
UPDATE THE 4D 
STATE OF THE 
FEATURE. THE STATE 
COVARIANCE MATRIX 
AND THE KALMAN 
FILTER GAIN 



S108 



S106 




UPDATE THE 4D 
STATE OF THE 
FEATURE. THE STATE 
COVARIANCE MATRIX 
AND THE KALMAN 
FILTER GAIN USING 

THE REFINED 
POSITION OF THE 
FEATURE 



S110 



ANOTHER 
TRACKED FEATURE 
IN THE NEW KEYFRAME AND 
THE PRECEDING 
KEYFRAME? 



NO 
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S140 



SET THE FIRST KEYFRAME IN THE 
SEQUENCE AS FRAME 1 OF THE FIRST 
TRIPLE 


p 


< 
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CONSIDER THE N( 
THE SEC 


EXT KEYFRAME IN 
3UENCE 



SI 32 



S134 



NO 



IS THE 
^NUMBER OF TRACKED" 
FEATURES WHICH ARE IN 
BOTH THE CURRENT KEYFRAME 
AND FRAME 1 OF THE 
TRIPLE LESS THAN A 
THRESHOLD? 



SI 36 



REMOVE FRAME 1 FROM 
THE TRIPLE AND SET 
THE CURRENT 
KEYFRAME AS FRAME 1 
OF A NEW TRIPLE 



YES 



S138 



NO 



IS THERE 
"at LEAST ONE KEYFRAME^ 
BETWEEN THE CURRENT 
KEYFRAME AND FRAME 1 
OF THE TRIPLE? 



YES 



SET THE KEYFRAME PRECEDING THE 
CURRENT KEYFRAME AS FRAME 3 OF 
THE TRIPLE 



REMOVE 
FRAME 3 FROM 
THE TRIPLE 
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S142 
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Fig. 9 
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S144 




SI 52 



SET THE MIDDLE 
KEYFRAME BETWEEN 
FRAME 1 AND FRAME 3 AS 
FRAME 2 OF THE TRIPLE 



SET FRAME 2 OF THE TRIPLE TO 
BE THE FRAME FROM THE 
MIDDLE PAIR OF KEYFRAMES 
WHICH CONTAINS THE HIGHEST 
NUMBER OF TRACKED 
FEATURES WHICH ARE ALSO 
PRESENT IN FRAME 1 AND 
FRAME 3 OF THE TRIPLE 



CONSIDER NEXT KEYFRAME OF THE 
KEYFRAMES AFTER FRAME 3 OF THE 
PRECEDING TRIPLE 



S154 



NO 



IS THE 
'NUMBER OF TRACKED 
FEATURES WHICH ARE IN 
BOTH THE CURRENT KEYFRAME 
^ND FRAME 3 OF THE PRECEDING 
TRIPLE LESS THAN A 
THRESHOLD? 
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YES 



SET THE KEYFRAME AS FRAME 1 OF 
A NEW TRIPLE 
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READ DATA FOR NEXT TRIPLE 
OF KEYFRAMES 



S180 
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READ DATA FOR NEXT REMAINING 
TRIPLE OF KEYFRAMES 



S200 




YES 



CALCULATE RELATIVE CAMERA 
TRANSFORMATIONS FOR EACH 
KEYFRAME LYING BETWEEN 
KEYFRAMES OF THE TRIPLE 



S204 



ADJUST THE CAMERA 
TRANSFORMATIONS FOR THE SET 
OF KEYFRAMES COMPRISING THE 
TRIPLE OF KEYFRAMES AND THE 
KEYFRAMES THEREBETWEEN 
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NO 



Fig. 12 
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CALCULATE A FEATURE POINT IN 3D 
FOR EACH TRACKED FEATURE WHICH 
IS PRESENT IN AT LEAST TWO OF THE 

THREE KEYFRAMES OF THE TRIPLE 



S220 



READ DATA FOR THE NEXT KEYFRAME 
WHICH IS BETWEEN THE KEYFRAMES 
OF THE TRIPLE 



S222 



LIST CORRESPONDENCES BETWEEN 
THE 3D FEATURE POINTS AND THE 
POSITIONS OF THE TRACKED 
FEATURES IN THE INTRA-TRIPLE 
KEYFRAME 



S224 



SET COUNTER = COUNTER + 1 



SELECT 3 CORRESPONDENCES AT 
RANDOM 



S226 



S228 



CALCULATE CAMERA 
TRANSFORMATIONS FOR THE INTRA- 
TRIPLE KEYFRAME USING THE 
SELECTED CORRESPONDENCES 



S230 



TEST THE CALCULATED 
TRANSFORMATIONS AGAINST EACH 
CORRESPONDENCE 



S232 



Fig 
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ARE THE 
TRANSFORMATIONS 
MORE ACCURATE THAN ANY 
PREVIOUSLY CALCULATED 
.FOR THE INTRA-TRIPLE. 
KEYFRAME? 



S234 



NO 



YES 



STORE THE CALCULATED 
TRANSFORMATIONS AND ASSOCIATED 
ACCURACY 



S236 



S238 




NO 



YES 



S240 



YES 



HAS ACCURACY 
OF CALCULATED 
TRANSFORMATIONS 
INCREASED IN THE PREVIOUS 
^^REDETERMINED NUMBER. 
OF ITERATIONS? 



NO 



S242 



DOES THE 
""ACCURACY OF THE STORED^ 
CAMERA TRANSFORMATIONS 
^EXCEED A PREDETERMINED 
THRESHOLD? 



NO 



YES 



Fig. 13 
(cont) 
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YES 
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DISCARD THE INTRA-TRIPLE 
KEYFRAME 



S244 



ANOTHER 
KEYFRAME WHICH IS 
BETWEEN THE KEYFRAMES 
OF THE TRIPLE? 
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NO 



Fig. 13 (con 



FEATURE POINT 1 



FEATURE POINT 2 



FEATURE POINT 3 



FEATURE POINT n 
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262 
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3D POSITION 
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2D POSITION IN 
INTRA-TRIPLE 
KEYFRAME 



Fig. 14 
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READ THE 2D POSITION OF EACH 
TRACKED FEATURE IN EACH OF THE 
KEYFRAMES WITHIN THE NEXT 
WINDOW 



CALCULATE AN ERROR 
MEASUREMENT VALUE FOR THE 
SUBSET OF KEYFRAMES CURRENTLY 
BEING CONSIDERED USING THE 
CALCULATED CAMERA 
TRANSFORMATIONS AND THE 
POSITIONS OF THE TRACKED 
FEATURES WITHIN THE KEYFRAMES 
AND IN 3D 



ADJUST THE CAMERA 
TRANSFORMATIONS AND THE 
POSITIONS OF THE TRACKED 
FEATURES IN 3D TO MINIMISE THE 
ERROR MEASUREMENT VALUE 
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CALCULATE CAMERA 
TRANSFORMATIONS FOR 
KEYFRAMES BETWEEN SETS BY 
EXTENDING THE SETS OF 
KEYFRAMES 



S290 



MERGE SETS OF KEYFRAMES 
WHICH MEET 



S292 





MERGE SETS OF KEYFRAMES 
WHICH DO NOT MEET 
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Fig. 19 
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DETERMINE A PROCESSING ORDER 
FOR THE ENDS OF THE SETS OF 
KEYFRAMES 



CONSIDER NEXT END KEYFRAME 



CONSIDER THE KEYFRAME IN THE 
GAP ADJACENT TO THE END 
KEYFRAME 



S300 



S302 



S304 



READ THE TRACKED FEATURES IN THE 
GAP KEYFRAME AND IDENTIFY 
CORRESPONDING 3D POINTS. IF 3D 

POINT DOES NOT EXIST FOR A 
FEATURE AND THE FEATURE IS ALSO 

PRESENT IN AT LEAST TWO 
KEYFRAMES OF THE SET. CALCULATE 
3D POINT 



S306 




DISCARD THE 
CALCULATED 
CAMERA 
TRANSFORMATION 
AND DELETE THE 

CURRENT END 
KEYFRAME FROM 
THE PROCESSING 
ORDER 



S312 



S314 



CURRENT KEYFRAME 

BECOMES PART OF 
THE SET. ADJUST THE 
CALCULATED CAMERA 
TRANSFORMATIONS 
FOR THE SET 



UPDATE PROCESSING 
ORDER 



S316 
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Fig. 20 
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Fig. 20 (con 



CONSIDER NEXT PAIR OF KEYFRAME 
SETS WHICH MEET 



S330 



REGISTER THE CALCULATED 
CAMERA TRANSFORMATIONS FOR 
THE KEYFRAMES ADJACENT THE 
BOUNDARY 



S332 




Fig. 22 
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CONSIDER ALL KEYFRAMES IN THE 
FIRST SET AND ALL KEYFRAMES IN 
THE SECOND SET 



IDENTIFY EACH TRACKED FEATURE 
WHICH IS IN AT LEAST TWO 
KEYFRAMES FROM EACH SET 



READ THE 3D POSITION OF EACH 
FEATURE POINT CORRESPONDING TO 
AN IDENTIFIED FEATURE WHICH HAS 

PREVIOUSLY BEEN CALCULATED 
USING THE KEYFRAMES IN THE FIRST 
SET 



READ THE 3D POSITION OF EACH 
FEATURE POINT CORRESPONDING TO 
AN IDENTIFIED FEATURE WHICH HAS 
PREVIOUSLY BEEN CALCULATED 
USING THE KEYFRAMES IN THE 
SECOND SET 



USE THE 3D POSITIONS OF 
CORRESPONDING FEATURE POINTS 

TO CALCULATE THE 
TRANSFORMATION BETWEEN THE 
COORDINATE SYSTEMS OF THE FIRST 
AND SECOND KEYFRAME SETS 
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CONSIDER NEXT GAP BETWEEN 
CONSECUTIVE PAIRS OF KEYFRAME 
SETS 
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DISCARD THE KEYFRAMES WITHIN 
THE GAP 
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MATCH FEATURES IN THE KEYFRAMES 
ADJACENT THE GAP 
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YES 



IS NUMBER 
OF MATCHED FEATURES 
LESS THAN A 
THRESHOLD? 
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NO 



USE THE KEYFRAMES IN THE SET 
CONTAINING THE FIRST KEYFRAME 
ADJACENT THE GAP TO CALCULATE A 
FEATURE POINT IN 3D FOR EACH 
FEATURE MATCHED IN THE 
KEYFRAMES ADJACENT THE GAP FOR 
WHICH A FEATURE POINT HAS NOT 
ALREADY BEEN CALCULATED 



USE THE KEYFRAMES IN THE SET 
CONTAINING THE SECOND KEYFRAME 
ADJACENT THE GAP TO CALCULATE A 
FEATURE POINT IN 3D FOR EACH 
FEATURE MATCHED IN THE 
KEYFRAMES ADJACENT THE GAP FOR 
WHICH A FEATURE POINT HAS NOT 
ALREADY BEEN CALCULATED 
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Fig. 25 
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USE THE MATCHED FEATURES TO 
REGISTER THE CAMERA 
TRANSFORMATIONS FOR THE 
KEYFRAMES ADJACENT THE GAP 
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Fig. 25 (con 
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